Source-linked AI summary
Large-Scale Adversarial Training for Vision-and-Language Representation Learning
Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, Jingjing Liu
TL;DR
Large-scale V+L models can overfit limited downstream labels, motivating adversarial training for more robust representation learning. VILLA applies adversarial training during pre-training and finetuning in multimodal embedding spaces, achieving new state-of-the-art results across six V+L tasks. The approach consistently improves evaluated benchmarks, while adversarial training remains time-consuming and the paper leaves comprehensive V+L attack and robustness analysis for future work.
Problem
Large-scale pre-trained V+L models can overfit when aggressively finetuned on limited downstream labeled data.
Method
VILLA performs task-agnostic adversarial pre-training and task-specific adversarial finetuning with perturbations in word and image-region embedding spaces, using “free” training and KL regularization.
Results
VILLA achieves new state-of-the-art performance across six V+L tasks, including VQA, VCR, NLVR2, Visual Entailment, Referring Expression Comprehension, and Image-Text Retrieval.
Takeaways & Limitations
Adversarial training in both pre-training and finetuning stages produces consistent performance boosts across the evaluated V+L benchmarks.
Takeaways & Limitations
The paper focuses on clean-data generalization and leaves thorough investigation of adversarial attacks and robustness for future work.
Abstract
from arXiv · showhide
We present VILLA, the first known effort on large-scale adversarial training for vision-and-language (V+L) representation learning. VILLA consists of two training stages: (i) task-agnostic adversarial pre-training; followed by (ii) task-specific adversarial finetuning. Instead of adding adversarial perturbations on image pixels and textual tokens, we propose to perform adversarial training in the embedding space of each modality. To enable large-scale training, we adopt the "free" adversarial training strategy, and combine it with KL-divergence-based regularization to promote higher invariance in the embedding space. We apply VILLA to current best-performing V+L models, and achieve new state of the art on a wide range of tasks, including Visual Question Answering, Visual Commonsense Reasoning, Image-Text Retrieval, Referring Expression Comprehension, Visual Entailment, and NLVR2.
1 Introduction
VILLA introduces adversarial training for vision-and-language representation learning through task-agnostic pre-training and task-specific finetuning. It perturbs multimodal embeddings rather than pixels or tokens and improves performance across six V+L tasks.
- Framework: VILLA combines task-agnostic adversarial pre-training with task-specific adversarial finetuning.The authors hypothesize that generalization learned during pre-training transfers to diverse downstream tasks, while finetuning adds task-specific gains.
- Framework: Adversarial perturbations are added to word embeddings and extracted image-region features instead of textual tokens and image pixels.Experiments report large performance gains on clean inputs from this embedding-level strategy.
- Framework: “Free” adversarial training reduces the extra computational cost of obtaining parameter gradients during adversarial training.KL-divergence-based regularization also encourages prediction confidence to remain close, promoting smoother embedding-space behavior.
- Evaluation: VILLA is evaluated by enhancing UNITER across six V+L tasks and is additionally applied to LXMERT for a generalizability test.The six UNITER tasks are VQA, VCR, NLVR2, Visual Entailment, Referring Expression Comprehension, and Image-Text Retrieval.
- Results: VILLA achieves new state-of-the-art results across six popular V+L tasks.Using standard bottom-up image features, UNITER-large improves from 74.02 to 74.87 on VQA and from 62.8 to 65.7 on VCR; ensemble VQA reaches 75.85.
2 Related Work
Related work spans multimodal pre-training architectures and earlier approaches to V+L representation learning. VILLA extends adversarial training into V+L pre-training and finetuning with perturbations in both image and word embedding spaces.
- Multimodal Pre-training: Early multimodal pre-training used two-stream Transformers, while later models commonly adopted single-stream designs that fuse modalities earlier.ViLBERT and LXMERT exemplify two-stream architectures; VL-BERT, VisualBERT, B2T2, Unicoder-VL, and UNITER exemplify single-stream models.
- V+L Representation Learning: Earlier V+L representation-learning work developed attention mechanisms, multimodal fusion, multi-step reasoning, object relations, and neural module networks.These approaches targeted stronger multimodal representations and compositional reasoning before multimodal pre-training became dominant.
- Adversarial Training: Adversarial training has primarily targeted image classification, with parallel work addressing language modeling, natural language understanding, and pre-training.The literature often treats robustness and clean-image accuracy as competing objectives.
- Adversarial Training: VILLA is presented as the first known study of adversarial training for V+L tasks, perturbing both image and word embedding spaces.The paper also studies incorporating adversarial training into both pre-training and finetuning.
3 Vision-and-Language Large-scale Adversarial Training
VILLA combines two-stage adversarial training with embedding-space perturbations and an efficient regularization strategy for vision-and-language models. It applies adversarial pre-training and task-specific finetuning across multimodal objectives and downstream tasks.
- Framework: VILLA has three key designs: adversarial pre-training and finetuning, embedding-space perturbations, and an enhanced adversarial training algorithm.The framework uses task-agnostic adversarial pre-training followed by task-specific adversarial finetuning.
- Framework: Pre-training learns universal image-text representations from paired images and text, while downstream finetuning adds a task-specific network over the joint [CLS] representation.Pre-training includes MLM, MRM, and ITM objectives; finetuning minimizes cross-entropy on supervised downstream data.
- Embedding-space perturbations: Adversarial perturbations modify image features and word embeddings one modality at a time, while positional embeddings and other multimodal components remain unchanged.Perturbation norms are constrained to preserve original semantics, and the approach operates in embedding space rather than image pixels or discrete tokens.
- Enhanced adversarial training: The adversarial objective combines clean cross-entropy, label-preserving adversarial loss, and KL-based regularization that keeps perturbed and clean prediction distributions close.The symmetric KL term targets prediction confidence in addition to preserving labels.
- Enhanced adversarial training: “Free” adversarial training accumulates parameter gradients across multiple PGD iterations and updates model parameters once, reducing the cost of large-scale training.This produces a K-times-larger virtual mini-batch while generating adversarial embeddings through constrained gradient ascent.
4 Experiments
Experiments evaluate VILLA across diverse vision-and-language tasks and compare its pre-training, finetuning, modality, algorithmic, and representation effects. VILLA improves over UNITER and LXMERT baselines, with gains observed across benchmarks and evidence of regularization and stronger multimodal attention.
- Experimental Setting: VILLA is evaluated on six downstream tasks using UNITER, with additional generalizability tests using LXMERT on VQA, GQA, and NLVR2.The evaluation covers VQA, VCR, NLVR2, Visual Entailment, Referring Expression Comprehension, and Image-Text Retrieval.
- Downstream Task Evaluation: +0.76 on VQA, +2.4 on VCR Q→AR, and +1.45 on NLVR2 are among VILLA-base’s gains over UNITER-base.Additional gains are +0.75 on SNLI-VE, +2.22/+0.70 on Flickr30k Image/Text Retrieval R@1, and +0.99 average on three referring-expression datasets.
- Pre-training vs. Finetuning: +1.15 points is gained by combining adversarial pre-training and finetuning, compared with +0.51 and +0.82 points from either stage alone.The ablation averages results over six evaluation tasks.
- Training Curves: Adversarially enhanced models develop growing performance gaps over standard UNITER as training steps increase, while VILLA quickly catches up on VQA after initially lagging.The reported curves are provided for multiple datasets, with similar trends observed on the remaining datasets.
- Attention Probing: VILLA achieves higher attention weights across ten visual coreference and relation categories, averaging 0.223 versus 0.195 for UNITER.The probing analysis measures attention between regions and phrases or between regions.
- LXMERT Generalization: VILLA-fine provides a +0.88 average performance boost across VQA, GQA, and NLVR2 with LXMERT and shows a regularization effect against overfitting.It achieves higher validation accuracy and lower training accuracy than LXMERT on VQA and GQA.
5 Conclusion
VILLA is presented as an adversarial-training framework for vision-and-language representation learning, applied across both pre-training and finetuning. The authors identify accelerating adversarial training as future work, while code is available.
- VILLA applies adversarial training in both pre-training and finetuning to improve vision-and-language representation learning.
- The authors report consistent performance boosts across all evaluated benchmarks.
- Accelerating adversarial training is identified as future work to make large-scale pre-training more feasible.
- Code for the LXMERT adversarial-training implementation is available online.
Broader Impact
VILLA incorporates adversarial training into both pre-training and finetuning and is intended to support future vision-and-language research. Its large-scale training requires substantial computational resources, with financial and environmental costs.
- VILLA incorporates adversarial training in both pre-training and finetuning stages.
- Web-scale image-text pre-training allows VILLA to absorb multimodal clues and capture multichannel signals.
- The method provides an instant performance boost during finetuning, according to the authors.
- Large-scale training demands substantial computational resources and can be costly financially and environmentally.
- The authors plan to release pre-trained models to facilitate future research and save environmental cost.
A Appendix
The supplementary material contains sections covering additional related work, experimental results, and downstream-task and implementation details.
- The supplementary material includes additional related work, experimental results, and downstream-task and implementation details.
A.1 Additional Related Work
The related-work section surveys adversarial-training improvements involving inter-sample interactions, extra unlabeled data, and faster training, while noting growing interest in vision-and-language adversarial examples.
- Prior adversarial-training work uses tripletwise metric learning and optimal transport to leverage inter-sample interactions.
- Other approaches exploit extra unlabeled training data or accelerate the training procedure.
- Adversarial examples have been studied primarily in images and only more recently in vision-and-language research.
A.2 Additional Results
Additional experiments show that VILLA improves task performance across question types and model sizes, while combining adversarial pre-training and finetuning yields further gains. Training curves and attention visualizations provide supplementary analyses of its behavior.
- VQA results: VILLA improves UNITER on all reported VQA question types.The detailed results are provided in Table 7.
- Training curves: The remaining three datasets show training curves with a trend similar to the three datasets presented in Figure 3a.Figure 5 contains the additional curves for VILLA and UNITER.
- Ablation results: +1.48 and +2.21 performance gains come from adversarial pre-training and finetuning, respectively, with larger models; combining both stages improves further.These results are reported for the 24-layer model on VQA and VCR.
- Additional pre-training: Adversarial pre-training from scratch brings further performance improvement on VQA.The paper describes these results as preliminary and leaves thorough investigation for future work.
- Attention visualization: Additional text-to-image attention visualizations compare VILLA and UNITER.These visualizations are presented in Figure 6.
A.3 Downstream Tasks and Implementation Details
The downstream evaluation covers classification, ranking, alignment, and probing tasks across vision-and-language benchmarks. Implementation uses shared prediction architectures where possible, task-specific objectives, and GPU-based mixed-precision distributed training.
- Downstream task formulations: VQA, GQA, and VCR predict answers from an image and question, while NLVR2 judges descriptions against image pairs.Visual Entailment predicts whether an image entails a sentence, and Referring Expression Comprehension evaluates image-region alignment.
- Task-specific modeling: Most tasks use the joint V+L embedding from [CLS] with an MLP, whereas Referring Expression Comprehension uses region-wise alignment scores.Image-Text Retrieval is trained as a ranking problem with triplet loss and hard negatives; other tasks use cross-entropy classification.
- Probing analysis: Visual coreference probing examines all 144 encoder attention heads against annotated image-region–noun-phrase links.The task tests whether visual coreference knowledge is captured by attention traces.
- Probing analysis: Visual relation probing uses Visual Genome, containing 1,531,448 relations, and samples at most 15,000 relation pairs per relation type.The analysis examines attention weights on ground-truth links after reducing relation-type imbalance.
- Implementation details: Experiments use PyTorch, Nvidia Apex mixed precision, V100 or Titan RTX GPUs, and Horovod with NCCL for large multi-node pre-training.Hyperparameter values are listed in Table 11.