PaperlayerView exact revision on arXiv

Source-linked AI summary

12-in-1: Multi-Task Vision and Language Representation Learning

Jiasen Lu, Vedanuj Goswami, Marcus Rohrbach, Devi Parikh, Stefan Lee

arXiv:1912.02315v2cs.CVcs.CLcs.LG

TL;DR

Vision-and-language research studies specialized tasks separately even though their visually grounded language skills overlap. The paper develops a single multi-task model across 12 datasets and reports lower model size with improved average performance, while using the framework to analyze joint training and initialize stronger task-specific models.

  • Problem

    Vision-and-language tasks are often studied with separate models despite overlapping language-to-visual-concept associations.

  • Method

    The paper trains a ViLBERT-based multi-task model jointly on 12 datasets across four task categories and uses a cleaned split to prevent cross-task leakage.

  • Results

    2.05 average-performance points improve over independent models while reducing parameters from ∼3 billion to ∼270 million; multi-task training improves 11 of 12 tasks.

  • Takeaways & Limitations

    Multi-task learning can outperform independent task training and serve as effective pretraining for state-of-the-art single-task models.

  • Takeaways & Limitations

    Failure cases mostly involve counting questions and difficult referring expressions requiring fine-grained recognition.

Abstract

from arXiv · show

Much of vision-and-language research focuses on a small but diverse set of independent tasks and supporting datasets often studied in isolation; however, the visually-grounded language understanding skills required for success at these tasks overlap significantly. In this work, we investigate these relationships between vision-and-language tasks by developing a large-scale, multi-task training regime. Our approach culminates in a single model on 12 datasets from four broad categories of task including visual question answering, caption-based image retrieval, grounding referring expressions, and multi-modal verification. Compared to independently trained single-task models, this represents a reduction from approximately 3 billion parameters to 270 million while simultaneously improving performance by 2.05 points on average across tasks. We use our multi-task framework to perform in-depth analysis of the effect of joint training diverse tasks. Further, we show that finetuning task-specific models from our single multi-task model can lead to further improvements, achieving performance at or above the state-of-the-art.

1. Introduction

Vision-and-language research often isolates specialized tasks despite shared visually grounded language skills. This work trains one multi-task model across 12 datasets, improving average performance while sharply reducing parameters.

  • Motivation: Specialized vision-and-language models remain fragmented across tasks, despite shared associations between language and visual concepts.The paper gives grounding “small red vase” and answering a color question about the same vase as an example of overlapping skills.
  • Motivation: Joint training can pool grounding supervision across tasks and reduce overfitting to individual datasets and metrics.
  • Approach: A single ViLBERT-based model is trained jointly on 12 datasets spanning four vision-and-language task categories.The training regime uses a dynamic scheduler, task-dependent input tokens, and hyper-parameter heuristics to handle differing dataset sizes and difficulties.
  • Results: 0.25 to 4.19 absolute points of improvement occurs on 11 out of 12 tasks with multi-task training.The model also sets a new state-of-the-art for 7 out of 12 tasks after task-specific fine-tuning.
  • Contribution: The paper systematically analyzes relationships among tasks and evaluates whether joint training improves performance over single-task training.
  • Results: ∼3 billion to ∼270 million parameters accompanies a 2.05-point average-performance improvement versus independent models.

2. Vision-and-Language Tasks

The paper organizes 12 vision-and-language datasets into four task groups requiring varied inputs, outputs, grounding granularity, and reasoning skills. It also identifies substantial cross-task image overlap and removes overlapping test images from training and validation splits.

  • Task groups: The 12 datasets cover VQA, image retrieval, referring expressions, and multi-modal verification.
  • Vocab-based VQA: Vocab-based VQA selects an answer from a fixed vocabulary for an image-question pair, using VQAv2, GQA, and Visual Genome QA.
  • Image Retrieval: Image retrieval selects the best-described image from a pool using a caption, with COCO and Flickr30K datasets.
  • Referring Expressions: Referring-expression tasks identify an image region from language ranging from noun phrases to multi-round dialogs, including RefCOCO, Visual7W, and GuessWhat.
  • Multi-modal Verification: Multi-modal verification judges statements against images, covering NLVR2 and SNLI-VE.NLVR2 requires a statement to be true for both images, whereas SNLI-VE classifies entailment, contradiction, or neutral.
  • Clean V&L Multi-Task Setup: About 11% average training-data reduction results from removing test images found in other tasks’ train/validation sets.This cleaned split prevents annotations from other tasks from leaking visual-grounding clues into evaluation while leaving test sets unchanged.

3. Approach

The approach builds a shared ViLBERT-based trunk with task-specific heads for diverse vision-and-language tasks, then uses task tokens and large-scale training strategies to coordinate learning across datasets. It modifies pretraining and introduces sampling, stopping, and curriculum strategies to address mismatched task sizes, difficulties, and convergence rates.

  • Base Architecture: ViLBERT uses parallel visual and linguistic transformer streams linked by co-attentional layers to exchange information between modalities.The default configuration has 6 visual and 12 linguistic transformer layers.
  • Pretraining: Pretraining masks overlapping visual regions together and omits masked multimodal modelling loss for negative captions.These changes reduce visual-information leakage and noise from unmatched caption samples.
  • Multi-Task Model: A shared ViLBERT trunk is jointly trained with task-specific heads across all tasks.The model learns shared parameters θs and task-specific parameters {θt} while minimizing loss across tasks.
  • Multi-Task Model: Task tokens let the shared architecture process the same grounded inputs differently for each task.The token is prepended to the query so task information can influence processing from the bottom up.
  • Task Outputs: The task heads support vocabulary-based VQA, image retrieval, referring-expression grounding, and multimodal verification with task-specific outputs.Outputs include multilabel answer scores, image-caption alignment scores, region matching scores, and classification probabilities.
  • Large-Scale Multitask Training: Dynamic stop-and-go addresses overtraining and forgetting caused by unequal dataset sizes and task difficulties during round-robin training.The method periodically stops tasks while allowing validation-based continuation, whereas the reported curriculum experiment found no curriculum superior when combined with other strategies.

4. Experiments and Results

Experiments show that joint training across vision-and-language tasks improves performance while substantially reducing model size, and that cross-group task interactions are uneven. Fine-tuning the unified model further improves results and reaches or exceeds state-of-the-art performance on many tasks.

  • Intra-group Multi-task Training: 0.37 to 4.54 absolute points improve for 11 of 12 tasks under intra-group multi-task training, with only SNLI-VE declining.The cleaned-data intra-group models average 68.72, exceeding the 67.25 average of full-data single-task models.
  • Representative Task Analysis: VQA benefits other representative tasks by +1.04% on average, whereas NLVR2 harms them by -1.36% on average while gaining +1.48% itself.NLVR2’s strongest reported negative interaction is -4.13% with retrieval task G2.
  • Representative Task Analysis: Cross-group gains are not simply additive: G4’s -4.36% effect on G2 with G3 can be moderated to +0.49% when combined with G1.The analysis evaluates representative task pairs and triplets using relative performance changes from single-task training.
  • Full Multi-task Results: 2.05 points higher average performance comes from the All-Tasks model than from same-data single-task models, improving 11 of 12 tasks.The average score rises from 67.03 to 69.08.
  • Full Multi-task Results: 12× fewer parameters are required by the All-Tasks model, reducing over 3 billion parameters to 270 million.The unified model retains the reported average-performance improvement while using substantially fewer parameters.
  • Multi-Task Learning as Pretraining: 69.51 average performance follows fine-tuning from the All-Tasks model, compared with 68.81 from group-task initialization, while task-specific fine-tuning surpasses state-of-the-art on 7 of 12 tasks.The reported fine-tuning comparisons use cleaned datasets for multi-task models and full datasets for the final individual-task comparison.

5. Analysis and Ablation Study

The analysis studies task tokens, training schedules, dynamic stop-and-go behavior, visual grounding consistency, and regularization in multi-task learning. Across these experiments, task-aware inputs and the proposed schedule improve performance and cross-task grounding consistency, while joint training mitigates overfitting.

  • Task tokens: 69.08 average performance with dataset-specific task tokens exceeds 68.52 with head-based tokens and 68.53 without task tokens.The results support task-aware feature embeddings even when tasks share an output space.
  • Dynamic stop-and-go: Updating more sparsely initially improves normalized group accuracy, but larger iteration gaps degrade performance.Smaller datasets enter stop states early, periodically returning to full training as the shared base model changes.
  • Visual grounding consistency: 64.64% visual grounding consistency for AT->ST exceeds 58.30% for AT and 55.40% for ST.The metric weights correctness by overlap between visual concepts in paired questions and referring expressions.
  • Regularization: Multi-task training improves validation scores without overfitting on SNLI-VE and Flickr Grounding, whose single-task training overfits quickly.Figure 4 presents training and validation curves for these two tasks.
  • Qualitative examples: Qualitative examples show the single model producing outputs for varied vision-and-language inputs spanning the trained task families.The examples include retrieval, question answering, grounding, dialog-based region prediction, verification, and image inference.

6. Related Work

Related work situates the paper within multi-task learning and unified vision-and-language architectures. Existing approaches still often require task-specific adaptation, while prior multi-task V&L studies differ in task coverage or output-selection procedures.

  • Multi-task learning: Multi-task learning has been studied across vision, language, and robotics, including analyses of relationships among tasks and layer-wise task routing.The paper connects its work to prior efforts that train one model across multiple tasks.
  • Vision and language: Recent vision-and-language architectures provide common pretrained backbones, but commonly remain task-specific models requiring separate fine-tuning.The related-work discussion notes that small datasets may make such fine-tuning prone to overfitting.
  • Multi-task V&L learning: Prior multi-task V&L work includes HDC, which uses hyperparameter search to select layer outputs, whereas this method does not require that search.The paper states that its method outperforms both cited prior approaches [36] and [33].
  • Multi-task V&L learning: The paper differs from concurrent work on 12 dialogue datasets by focusing on a variety of vision-and-language tasks.Only two datasets in the concurrent work contain images, according to the passage.

7. Conclusion

The paper presents a large-scale multi-modal multi-task training framework with a scheduling method for tasks of different sizes and difficulties. Experiments across 12 vision-and-language datasets find gains over independent training and benefits for single-task model pretraining.

  • Conclusion: The framework explores relationships among 12 vision-and-language datasets using a single multi-task model.The model is described as outperforming 12 single-task models.
  • Conclusion: The training regime introduces task scheduling intended to help avoid over- or under-training tasks with differing sizes or difficulties.This scheduling approach is presented as part of the large-scale multi-modal multi-task learning framework.
  • Conclusion: Multi-task training produces significant gains over independent task training in the reported experiments.The conclusion states this at the paper level without restricting the claim to a single dataset.
  • Conclusion: Multi-task learning is an effective pretraining step for training state-of-the-art single-task models.The conclusion identifies this as a further outcome of the framework.

8. Supplementary

The supplement describes dataset cleaning, pretraining modifications, the shared ViLBERT architecture, and implementation details for multi-task learning. Cleaning removes test-image overlaps from training and validation data, while the pretraining changes reduce visual leakage and negative-sample noise.

  • Datasets: 13.02% of images are removed on average during cleaning, which excludes original test images from all training and validation sets.The original datasets' test sets remain unmodified.
  • Datasets: The cleaned dataset counts images rather than training samples, because tasks assign different numbers of samples to each image.Training-sample details are provided separately in Table 8.
  • Pretraining: The supplement details modifications to the base ViLBERT pretraining procedure.These modifications are presented before the architecture and implementation details.
  • Pretraining: Masking visual regions with overlap above 0.4 IoU reduces leakage from neighboring visual embeddings and forces greater reliance on language.The modification masks overlapping regions in addition to the selected region.
  • Architecture: The model shares a ViLBERT base across four task groups: Vocab-Based VQA, Image Retrieval, Refer Expression, and Multimodal Verification.Task-specific heads are jointly trained for these groups.

8.4. Implementation Details

The implementation uses a shared multi-task architecture with task-aware inputs and a dynamic stop-and-go scheduler to manage datasets that differ in size and difficulty. Experiments analyze training behavior, including concept drift and task interactions.

  • Training setup: The training setup uses ResNeXT-152 Faster-RCNN image features, BERT initialization, AdamW, linear warmup and decay, and 40K iterations on 8 NVIDIA V100 GPUs for 5 days.
  • Architecture: The model augments each query with a task token to learn task-aware feature embeddings.The architecture diagram identifies task-token augmentation as part of the multi-task design.
  • Training dynamics: RefCOCO+ validation accuracy shows sharp concept-drift drops during sparse stop-mode updates before go mode is reactivated.
  • Scheduling: A dataset-size gap of up to 16:1 occurs between VG QA and ReferCOCOg, while active training time also differs substantially among similarly sized datasets.The analysis attributes these differences to dataset difficulty as well as size.
  • Task interactions: Removing G4 improves MT scores by 0.48% over AT, indicating negative interactions from G4 during joint training and fine-tuning.The paper leaves removing negative interactions between tasks as an open issue.

8.6. Comparison with other SOTA

The comparison evaluates the proposed single-task and multi-task-to-single-task models against recent vision-and-language systems using different pretraining data. The authors report comparable performance with less pretrained data, while the supplied passages also describe evaluation protocols and grounding-consistency analysis.

  • Comparison with SOTA: OursST and OursAT->ST are compared with ViLBERT, Unicoder-VL, VisualBERT, LXMERT, and UNITER across the full datasets.
  • Pretraining data: The proposed models use Conceptual Caption pretraining, whereas competing methods combine different out-of-domain or in-domain datasets.The passage notes that in-domain pretraining usually gives better downstream performance because it reduces domain transfer.
  • Comparison with SOTA: The authors achieve comparable performance with less pretrained data than several recent methods.
  • Evaluation: VQA and GQA use test-dev splits, Retrieval COCO and Flickr30k use their respective 1K test splits, and NLVR2 uses testP.Other datasets are evaluated on their respective test splits.
  • Grounding consistency: Multi-task visual grounding consistency tests whether shared models produce more consistent visual groundings than independent task-specific models.It pairs VQA questions and referring expressions associated with the same images and evaluates correct answers and localization overlap.

8.9. Qualitative Results

Qualitative examples show one OurAT model operating across diverse vision-and-language tasks, including retrieval, question answering, grounding, dialog-based region selection, and multimodal validation. Reported failure cases concentrate on counting and fine-grained referring expressions.

  • Qualitative results: OurAT handles short and long reasoning questions, image retrieval, pointing tasks, referring expressions, and multimodal validation in qualitative examples.
  • Failure cases: Failure cases mostly involve counting questions and difficult referring expressions or phrases requiring fine-grained recognition.

8.10. Attention Visualizations

Attention visualizations examine image–sentence grounding and task-dependent attention patterns in pretrained and multi-task models. The visualizations associate attended image regions with words and show different sentence self-attention patterns across tasks.

  • Multi-task attention: OurAT uses sentence-to-sentence, sentence-to-image, image-to-sentence, and image-to-image attention distributions in task-specific visualizations.
  • Task-dependent attention: Different tasks produce significantly different sentence-to-sentence self-attention patterns in the multi-task model.
  • Qualitative analysis: The attention visualizations support examining whether a single model behaves consistently across the diverse tasks represented in the qualitative examples.
  • Pretrained attention: The pretrained model associates attended image patches with visual words such as “boy,” “mom,” “pet,” “white,” and “sheep.”The reported visualizations show meaningful grounding for boy, sheep, white, and pet.
  • Sentence attention: The multi-task model learns previous-word, bag-of-words, and next-word attention patterns in sentence self-attention.A specific visualization highlights previous-word attention in Layer 2, Head 0.
  • Visualization conventions: Qualitative attention figures include dashed orange boxes for referring-expression outputs regardless of task.
Loading 1912.02315v2…