Source-linked AI summary
Multi-task Learning of Hierarchical Vision-Language Representation
Duy-Kien Nguyen, Takayuki Okatani
TL;DR
Vision-language tasks are difficult to learn jointly because they are commonly trained separately, making relations and knowledge transfer across tasks unclear. The paper uses a shared hierarchical representation learned from diverse datasets, with task-specific decoders operating at corresponding hierarchy levels. It reports improvements over prior single-task methods across image caption retrieval, VQA, and visual grounding.
Problem
Vision-language tasks are often studied independently, leaving relations among tasks and transfer of task knowledge unclear; dedicated multi-task datasets are also unavailable for arbitrary task combinations.
Method
A shared encoder built from stacked Dense Co-attention layers learns hierarchical visual-linguistic representations, while task-specific decoders use levels matched to task complexity and training alternates across datasets.
Results
The method outperforms previous individually trained methods on image caption retrieval, visual question answering, and visual grounding.
Takeaways & Limitations
The learned shared hierarchical representation generalizes across the three evaluated vision-language tasks and datasets.
Takeaways & Limitations
Creating dedicated datasets with multiple objectives is not generally feasible, and using shared validation data can cause training-test contamination across tasks.
Abstract
from arXiv · showhide
It is still challenging to build an AI system that can perform tasks that involve vision and language at human level. So far, researchers have singled out individual tasks separately, for each of which they have designed networks and trained them on its dedicated datasets. Although this approach has seen a certain degree of success, it comes with difficulties of understanding relations among different tasks and transferring the knowledge learned for a task to others. We propose a multi-task learning approach that enables to learn vision-language representation that is shared by many tasks from their diverse datasets. The representation is hierarchical, and prediction for each task is computed from the representation at its corresponding level of the hierarchy. We show through experiments that our method consistently outperforms previous single-task-learning methods on image caption retrieval, visual question answering, and visual grounding. We also analyze the learned hierarchical representation by visualizing attention maps generated in our network.
1. Introduction
Vision-language tasks are usually studied separately despite shared structure and possible hierarchical relations. The paper proposes joint learning of a shared hierarchical representation from diverse datasets, reporting gains across three tasks.
- The paper targets limited progress in multi-task vision-language learning caused by differences in task inputs, outputs, and complexity.
- Dedicated multi-task datasets can support joint training, but their absence and narrow task coverage limit generalization to other tasks or datasets.
- The proposed framework learns a shared hierarchical vision-language representation from diverse data sources using stacked Dense Co-attention layers and task-specific decoders.Tasks use representations from hierarchy levels corresponding to their complexity, and training alternates across tasks and datasets.
- Across image caption retrieval, visual question answering, and visual grounding, the method outperforms previous methods trained on individual tasks and datasets.
2. Related Work
Prior vision-language research developed shared multimodal representations and transfer methods largely one task at a time. Early multi-task studies demonstrated promise but depended on datasets with supervision shared across tasks.
- Vision-language representation studies generally learn supervised shared image-text representations for a single task at a time.
- Transfer-learning approaches reuse networks trained on source tasks to enrich representations for target vision-language tasks.
- Earlier multi-task work jointly refined representations or trained related tasks such as scene graph generation, object detection, image captioning, VQA, and visual question generation.
- These multi-task approaches rely strongly on datasets that provide supervision for multiple tasks on a shared input.
3. Learning Vision-Language Interaction
The model maps image-sentence pairs to task outputs through a shared hierarchical encoder and task-specific decoders. Dense Co-attention layers update visual and linguistic features, while decoders summarize or score representations for retrieval, VQA, and grounding.
- 3.1. Problem Formulation: Each task estimates an output from an image-sentence pair, with task-specific outputs for answer scores, caption relevance, or phrase-to-region assignments.
- 3.2. Shared Encoder: Stacked Dense Co-attention layers incrementally fuse image and sentence features, using symmetric attention between image regions and sentence words.
- 3.2. Shared Encoder: The network uses one shared encoder and multiple task-specific decoders, with each decoder attached to a selected intermediate representation.
- 3.3.1 Image Caption Retrieval: Image-caption retrieval summarizes attended image and sentence features, then computes their relevance with a scoring layer.
- 3.3.2 Visual Question Answering / 3.3.3 Visual Grounding: The VQA decoder derives attention-based image and sentence summaries and predicts scores for predefined answers, while visual grounding pools phrase words and scores phrase-region pairs.
4. Training on Multiple Tasks
The network is trained across multiple tasks by switching between task-specific updates rather than minimizing a combined loss simultaneously. Training uses curriculum learning, task-specific scheduling, and individually selected hyperparameters and attachment layers.
- Task-switching schedule: Task updates alternate because the tasks have different inputs, preventing simultaneous minimization of a summed loss.The network is trained on one selected task at a time, then switches to another task.
- Task-switching schedule: Curriculum learning expands training from single tasks to task pairs and finally all tasks.When multiple tasks are trained, periodical switching uses task-specific update counts before each switch.
- Choosing Layers Best Fit for Tasks: Each task's decoder is attached to a selected shared-encoder layer, with learning-rate steps, iteration counts, and batch sizes chosen by grid search on individual tasks.The selected layer and batch size remain fixed in subsequent joint training.
- Training algorithm: Algorithm 1 organizes training around task-specific decoders, shared-encoder sub-networks, sampled input-output pairs, and iteration over the active task set.The decoder for task i attaches at layer l(i), whose encoder output is represented as E_l(i).
5. Experiments
Experiments evaluate joint learning across image caption retrieval, visual question answering, and visual grounding, using multiple datasets and controlled training procedures. Joint learning generally improves performance, with the strongest results from combining all three tasks, while qualitative visualizations show task-specific attention and alignment behaviors.
- 5.3. Effects of Joint Learning of Multiple Tasks: Joint learning of all three tasks yields the best performance, while joint learning of two tasks performs better or comparably to single-task learning.Table 1 reports Accuracy for VQA and Recall@1 for visual grounding and image caption retrieval on MS-COCO.
- 5.3. Effects of Joint Learning of Multiple Tasks: 72.07 versus 67.16 for Flickr30k image annotation, compared with 70.43 versus 69.05 for MS-COCO, showing larger joint-learning gains on the smaller retrieval dataset.The comparison concerns joint learning with VQA and visual grounding; gains for VQA and visual grounding from adding image caption retrieval are smaller on Flickr30k than MS-COCO.
- 5.4. Full Results on Test Sets: Image caption retrieval is comparable to S-E Model on MS-COCO but exceeds its best published Flickr30k result by about 9.5% on all six evaluation criteria.The authors attribute the Flickr30k improvement to leveraging other tasks to compensate for limited image-caption retrieval training data.
- 5.4. Full Results on Test Sets: VQA surpasses DCN [29] by approximately 0.7% on both test-dev and test-standard, with gains across Other, Number, and Yes/No question types.The Number accuracy is reported as on par with the specialized Counting Module.
- 5.4. Full Results on Test Sets: Visual grounding performs below RTP on R@5 and R@10 but achieves a much better result on the hardest R@1 metric using only phrase-box correspondences.The method does not use box size, color, segmentation, or pose-estimation information used by previous studies [40].
- 5.5. Qualitative Evaluation: Qualitative visualizations show VG aligning entities with image regions, ICR attending more broadly to related entities and relations, and VQA narrowing attention to answer-relevant regions and words.The ICR decoder assigns 3.573e-07 to an inconsistent image-question pair and 0.387 to a consistent pair, matching the corresponding VQA answers “No” and “Yes”.
6. Summary and Conclusion
The paper presents a multi-task vision-language framework with a shared hierarchical encoder and task-specific decoders operating at corresponding hierarchy levels. Experiments across three major tasks show that the learned representation generalizes well across tasks.
- The framework combines a shared encoder that fuses visual and linguistic representations hierarchically with task-specific decoders assigned to corresponding levels.
- Experiments on three major vision-language tasks and their datasets demonstrate the approach's effectiveness.
- The shared hierarchical representation learned by the encoder generalizes well across tasks.