Source-linked AI summary
MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuning
Zhiyang Xu, Ying Shen, Lifu Huang
TL;DR
Instruction tuning had shown zero-shot promise in NLP, but its use for vision and multimodal tasks remained underexplored. The paper introduces MULTIINSTRUCT and fine-tunes OFA on its multimodal tasks, also testing transfer from text-only instructions; results report strong zero-shot performance on unseen multimodal tasks and reduced sensitivity to instruction variation.
Problem
Instruction tuning had shown zero-shot promise in NLP, but its use for vision and multimodal tasks had not been extensively explored.
Source: §1 · p. 1Method
The paper introduces MULTIINSTRUCT, a 62-task multimodal instruction benchmark, fine-tunes OFA on it, and tests transfer strategies using NATURAL INSTRUCTIONS.
Source: §1 · p. 1Results
Takeaways & Limitations
Fine-tuning on diverse tasks and instructions is associated with reduced sensitivity to variations in task instructions.
Source: §1 · p. 2, §8 · p. 9MULTIINSTRUCT targets English-language tasks and mainly focuses on vision-language tasks.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Instruction tuning, a new learning paradigm that fine-tunes pre-trained language models on tasks specified through instructions, has shown promising zero-shot performance on various natural language processing tasks. However, it has yet to be explored for vision and multimodal tasks. In this work, we introduce MUL-TIINSTRUCT, the first multimodal instruction tuning benchmark dataset that consists of 62 diverse multimodal tasks in a unified seq-to-seq format covering 10 broad categories. The tasks are derived from 21 existing open-source datasets and each task is equipped with 5 expert-written instructions. We take OFA as the base pre-trained model for multimodal instruction tuning, and to further improve its zero-shot performance, we explore multiple transfer learning strategies to leverage the large-scale NATURAL INSTRUCTIONS dataset. Experimental results demonstrate strong zero-shot performance on various unseen multimodal tasks and the benefit of transfer learning from a text-only instruction dataset. We also design a new evaluation metric - Sensitivity, to evaluate how sensitive the model is to the variety of instructions. Our results indicate that fine-tuning the model on a diverse set of tasks and instructions leads to a reduced sensitivity to variations in instructions for each task.
1 Introduction
Instruction tuning had shown zero-shot promise in NLP, but its use for vision and multimodal tasks remained underexplored. The paper introduces MULTIINSTRUCT and tests multimodal instruction tuning and transfer from text-only instructions, reporting strong performance on unseen multimodal tasks and reduced sensitivity to instruction variation.
1 Introduction
- Instruction tuning had improved zero-shot NLP performance, but its generalization to vision and multimodal tasks had not been extensively explored.The paper asks whether instruction tuning can improve vision-language models' generalizability on multimodal and vision tasks.Source: §1 · p. 1
- MULTIINSTRUCT provides 62 tasks across 10 categories, with five expert-written instructions per task and a unified sequence-to-sequence format.The format represents text, images, instructions, and bounding boxes in the same token space.Source: §1 · p. 1
- Fine-tuning OFA on MULTIINSTRUCT demonstrates strong zero-shot performance on unseen multimodal tasks, with further improvement possible through text-only instruction data.The paper explores mixed and sequential instruction tuning to leverage NATURAL INSTRUCTIONS.Source: §1 · pp. 1–2
- Sensitivity measures response to instruction variation; results report reduced OFA sensitivity with instruction tuning and further reductions as tasks and per-task instructions increase.Transfer from the larger text-only instruction dataset also reduces sensitivity.Source: §1 · p. 2
2 Related Work
Prior work developed unified multimodal pretraining and several language-model tuning strategies, including instruction tuning for text-only tasks. The paper places multimodal instruction tuning in a comparatively underexplored area.
2 Related Work
- Unified multimodal pretraining frameworks handle cross-modal and unimodal tasks; OFA extends unified text and visual tokens to image generation.VL-T5 uses a unified text-generation objective conditioned on multimodal inputs.Source: §2 · p. 2
- Prompt tuning, in-context learning, and instruction tuning were among strategies developed to improve language-model generalizability and adaptivity.NATURAL INSTRUCTIONS is described as a meta-dataset with diverse tasks, human-authored definitions, things to avoid, and demonstrations.Source: §2 · p. 2
- Instruction-tuning studies had succeeded in text-only domains, but vision and multimodal tasks had not been extensively explored.Prior work cited in the section includes zero-shot performance on unseen dialogue tasks.Source: §2 · p. 3
3 MULTIINSTRUCT
MULTIINSTRUCT builds a multimodal task collection from existing and derived tasks, equips tasks with expert-written instructions, and formats them as sequence-to-sequence generation problems.
3.1 Multimodal Task and Data Collection
- The dataset combines 34 existing tasks with 28 derived tasks, yielding 62 tasks across 10 categories and separating training from challenging, nonoverlapping evaluation tasks.Derived tasks vary input and output requirements; task instances come from open-source datasets or are extracted or reformulated from existing instances.Source: §3.1 · p. 3
3.2 Task Instruction Creation
- Instructions are task-describing templates with placeholders for input information, while images may be supplied separately to the model.The <OPTION> placeholder is used in classification tasks; <TEXT>, <REGION>, and <OPTION> can carry instance information.Source: §3.2 · p. 3
- Two expert annotators iteratively draft, review, and reconcile instructions, producing five instructions for each training and evaluation task.They receive task goals, input formats, and example instances; final review filters unrepresentative or overlapping tasks.Source: §3.2 · pp. 3–4
3.3 Multimodal Instruction Formatting
- Text, image, and bounding-box information are represented in a unified vocabulary, and all tasks become natural-language sequence-to-sequence generation problems.Text uses bytepair encoding, images use discrete VQ-GAN tokens, and bounding-box coordinates are discretized into location tokens.Source: §3.3 · p. 4
- For classification, [Options] marks the option field, |||| separates choices, and the model generates one option from the list.The input combines the filled instruction with the instance information.Source: §3.3 · pp. 4–5
4 Problem Setup and Models
The experiments fine-tune OFA on instruction tasks and evaluate its zero-shot capabilities on tasks that do not overlap with OFA pretraining or the MULTIINSTRUCT training set. They also test mixed and sequential transfer from NATURAL INSTRUCTIONS.
4.1 Problem Setup
- Each instruction task has examples pairing input information with outputs and a set of five expert-written instructions; input information fills the instruction placeholders.Inputs may include text, images, regions, and options.Source: §4.1 · p. 5
- OFA encodes instructions and instance information with an optional image, then decodes outputs; training instances are mixed and shuffled with a randomly sampled instruction per batch.Evaluation tasks are kept separate from both OFA pretraining tasks and MULTIINSTRUCT training tasks.Source: §4.1 · p. 5
4.2 Transfer Learning from NATURAL INSTRUCTIONS
- The transfer experiments use 832 English NATURAL INSTRUCTIONS tasks, either mixing their instances with MULTIINSTRUCT or tuning sequentially.In mixed tuning, each NATURAL INSTRUCTIONS task has one instruction, while a MULTIINSTRUCT instruction is randomly sampled from five.Source: §4.2 · p. 5
- Sequential tuning first trains OFA on NATURAL INSTRUCTIONS and then fine-tunes it on MULTIINSTRUCT to adapt instruction learning to multimodal tasks.The first stage uses all instances in English-language tasks.Source: §4.2 · p. 5
5 Experimental Setup
The setup evaluates zero-shot transfer on nine unseen multimodal tasks and 20 NLP tasks, using performance across five task instructions. It also defines Sensitivity to measure consistency across instruction variants.
Evaluation metrics
- Performance is measured with accuracy for classification and ROUGE-L for generation, with five instruction variants evaluated per task.The study reports mean, maximum, and standard deviation across the five experiments, plus aggregated performance across multimodal and NLP tasks.Source: §5 · pp. 5–6
Sensitivity metric
- Sensitivity measures whether a model produces consistent results when instruction wording varies but the intended task stays the same.It is based on the standard deviation and mean of performance across instructions, using metrics such as accuracy or ROUGE-L.Source: §5 · p. 6
Evaluation datasets
- Nine unseen multimodal tasks and 20 NLP tasks from NATURAL INSTRUCTIONS form the evaluation set.The multimodal tasks span Commonsense Reasoning, VQA, and Miscellaneous groups.Source: §5 · p. 6
Compared approaches
- The comparison includes pretrained OFA, OFA fine-tuned on MULTIINSTRUCT, a task-name control, and approaches using NATURAL INSTRUCTIONS.The text-only dataset is used alone or through mixed and sequential transfer strategies.Source: §5 · p. 6
6 Results and Discussion
Across unseen multimodal tasks, MULTIINSTRUCT instruction tuning improves OFA’s zero-shot performance, while broader task and instruction coverage is associated with improved performance and lower sensitivity. Transfer from text-only instructions offers mixed performance effects but can reduce sensitivity.
6.1 Effectiveness of Instruction Tuning on MULTIINSTRUCT
- MULTIINSTRUCT fine-tuning improves OFA’s zero-shot performance across unseen tasks and metrics, and outperforms task-name prompting.The authors attribute the gain mainly to instructions rather than multi-task training.Source: §6.1 · pp. 6–7, Table 1 · p. 7, Table 2 · p. 7
6.2 Impact of Transfer Learning from NATURAL INSTRUCTIONS
- Fine-tuning on NATURAL INSTRUCTIONS alone degrades multimodal performance on almost all tasks; transfer methods do not significantly outperform Mixed, but reduce standard deviation on 6 of 9 tasks.The authors suggest reduced attention from text tokens to image tokens may help explain the degradation after text-only fine-tuning.Source: §6.2 · p. 7
6.3 Impact of Increasing Multimodal Instruction Task Clusters
- Adding multimodal task clusters improves mean and maximum aggregated performance while decreasing sensitivity.Lower sensitivity indicates more consistent results despite instruction-wording variations.Source: Figure 3 · p. 7, §6.3 · p. 8
6.4 Effect of Diverse Instructions on Instruction Tuning
- Fine-tuning with five instructions per task yields higher aggregated performance and lower sensitivity than using one fixed instruction template.The authors suggest exploring crowdsourcing or automatic generation for still more diverse instructions.Source: §6.4 · p. 8, Table 3 · p. 8
6.5 Effect of Fine-tuning Strategies on Model Sensitivity
- Instruction-tuned models are less sensitive than original OFA, and transfer from NATURAL INSTRUCTIONS further reduces sensitivity across unseen multimodal tasks.Figure 4 reports average sensitivity across those tasks, with lower values indicating better consistency.Source: §6.5 · p. 8, Figure 4 · p. 8
7 Zero-Shot Performance on NLP Tasks
The paper tests multimodal instruction tuning on 20 unseen NLP tasks. OFAMultiInstruct outperforms pretrained OFA, while OFAMixedInstruct achieves the best aggregate performance across multimodal and NLP tasks and has lower sensitivity.
Zero-shot NLP performance
- On 20 NLP tasks, OFAMultiInstruct outperforms OFA despite the difference in modality between the tuning and evaluation datasets.OFANaturalInstruct achieves the best NLP-task performance, while OFAMixedInstruct better preserves NATURAL INSTRUCTIONS gains than OFASeqInstruct.Source: §7 · p. 8, Table 4 · p. 9
Overall results
- OFAMixedInstruct achieves the best aggregated performance across multimodal and NLP tasks and has lower sensitivity to instruction wording.The authors identify it as the most promising approach.Source: §7 · p. 9, Table 4 · p. 9
8 Conclusion
The paper introduces MULTIINSTRUCT for multimodal instruction tuning and reports improved zero-shot performance on unseen multimodal tasks, benefits from text-only instruction data, and reduced sensitivity to instruction wording.
8 Conclusion
- MULTIINSTRUCT is a benchmark covering a wide variety of vision and multimodal tasks, each paired with multiple expert-written instructions; finetuning OFA on it significantly improves zero-shot performance on various unseen multimodal tasks.Source: §8 · p. 9
- Fine-tuning OFA on MULTIINSTRUCT significantly improves zero-shot performance on unseen multimodal tasks, while transfer from NATURAL INSTRUCTIONS also benefits performance.The paper also reports that models become less sensitive to instruction variations after fine-tuning on diverse tasks and instructions.Source: §8 · p. 9
- Sensitivity evaluates how model performance varies with instruction wording, and the authors report reduced sensitivity after diverse instruction tuning.The reported metric concerns variations in instructions for each task.Source: §8 · p. 9
Limitations
The dataset and experiments have defined scope boundaries: the dataset is English-focused and mainly vision-language, while experiments rely on OFA and a within-task sensitivity measure.
Limitations
- The dataset targets English tasks and mainly focuses on vision-language, leaving multilingual tasks and modalities such as audio and video for future work.The authors also note that the 62 tasks and their associated instructions remain limited, suggesting crowd-sourcing or automatic generation and augmentation as possible ways to expand them.Source: Limitations · p. 9
Limitations of Experiments and Evaluation
- Experiments use OFA as the only baseline, and the authors call for comparisons with stronger publicly available multimodal models and different model sizes.They also identify alternative architectures and training losses as possible ways to better use text-only instruction datasets.Source: Limitations of Experiments and… · p. 9
- Sensitivity measures performance variation across instructions for the same task, but does not yet assess instruction understanding across different tasks.The authors identify inter-task sensitivity as a broader factor for future metric development.Source: Limitations of Experiments and… · p. 9
A Tasks Defined in MULTIINSTRUCT
The appendix documents MULTIINSTRUCT’s task and dataset statistics, evaluation task definitions, comparison approaches, and training setup.
A Tasks Defined in MULTIINSTRUCT
- Table 5 summarizes input/output modality distributions for MULTIINSTRUCT training and evaluation tasks, Table 6 reports detailed statistics separately, and Tables 7–9 list 62 included tasks with one example instruction each.Source: A Tasks Defined in MULTIINSTRUCT · p. 12
B.1 Multimodal Evaluation Datasets
- Evaluation tasks include reading image text, locating image regions, image-based commonsense reasoning, visual entailment, and visual and set-theoretic reasoning.The suite also includes text extraction from images, dialogue grounded in an image, and disaster-type classification.Source: §B.1 · pp. 12–13
- Visual Text Extraction requires extracting text appearing in an image, while Visual Dialogue uses an image and dialogue history and Disaster Type Classification predicts a disaster type from an image.Visual Text Extraction is derived from Hateful Memes.Source: §B.1 · p. 13
B.2 NLP Evaluation Tasks
- The NLP evaluation uses 20 tasks from NATURAL INSTRUCTIONS’ default test split, prepending each task’s Definition as an instruction to every input.The listed tasks include classification, entailment, question answering, and text generation tasks.Source: §B.2 · p. 13
B.3 Approaches for Comparison
- The comparison approaches include original OFA, task-name prompting, MULTIINSTRUCT-only tuning, and NATURAL INSTRUCTIONS-only tuning.The original OFA is the untuned pretrained model; the task-name approach prepends the task name rather than using the created instructions.Source: §B.3 · p. 13
- For a fair comparison, the NATURAL INSTRUCTIONS-only baseline is evaluated with instruction templates stripped of special tokens absent from its training data.The authors state this avoids bias toward models exposed to those tokens during training.Source: §B.3 · p. 13
- The mixed approach fine-tunes OFA on NATURAL INSTRUCTIONS and MULTIINSTRUCT together, while the sequential approach fine-tunes on those datasets in sequence.The supplied description identifies both datasets for the mixed setup and the sequential setup.Source: §B.3 · pp. 13–14
B.4 Training Details
- Training used a maximum input length of 1024 tokens and target length of 512, with three epochs on eight A100 GPUs at batch size 8 per GPU and learning rate 1e-05.The experiments were run once, using float16.Source: §B.4 · p. 14
C Attention Analysis
To investigate why fine-tuning on a text-only instruction dataset can lower zero-shot performance, the authors examine OFA’s text-to-image attention on three unseen multimodal tasks.
C Attention Analysis
- OFA’s encoder has 12 self-attention layers, and the first layer receives concatenated image and text embeddings of lengths I and T, respectively.Source: C Attention Analysis · p. 14
- For each layer, the authors sum each text query’s attention scores over image keys, then average across text states and attention heads to obtain a text-to-image attention score.Source: C Attention Analysis · p. 14
- Across three unseen tasks, OFANaturalInstruct has lower text-to-image attention than other models in every encoder layer, especially the first two; this may explain its zero-shot decline after text-only fine-tuning.The tasks are Text VQA, Visual Entailment, and Visual Text Extraction.Source: C Attention Analysis · p. 14, Figure 5 · p. 15