Source-linked AI summary
Multimodal Prompting with Missing Modalities for Visual Recognition
Yi-Lun Lee, Yi-Hsuan Tsai, Wei-Chen Chiu, Chen-Yu Lee
TL;DR
The paper addresses multimodal recognition when modalities are missing across samples or learning phases, alongside the computational burden of finetuning large transformers. It introduces missing-aware prompts for parameter-efficient adaptation and reports effectiveness across varied missing-modality cases while using less than 1% learnable parameters. Prompt placement and configuration are also analyzed, with performance depending on the prompting design and dataset setting.
Problem
Multimodal learning must handle modality-incomplete data during training or testing while avoiding the computational and generalization constraints of finetuning increasingly large transformers.
Method
The method freezes the pretrained multimodal transformer except task-specific layers and trains missing-aware prompts conditioned on each missing-modality case.
Results
The framework improves performance across various missing-modality cases while requiring less than 1% of the entire model’s parameters, with prompt placement and configuration affecting results.
Takeaways & Limitations
Missing-aware prompt learning provides a plug-in way to adapt multimodal transformers to general missing-modality scenarios without heavy model finetuning.
Takeaways & Limitations
Input-level prompting can be sensitive to prompt length and input length, with longer prompts potentially hurting performance on shorter-text datasets.
Abstract
from arXiv · showhide
In this paper, we tackle two challenges in multimodal learning for visual recognition: 1) when missing-modality occurs either during training or testing in real-world situations; and 2) when the computation resources are not available to finetune on heavy transformer models. To this end, we propose to utilize prompt learning and mitigate the above two challenges together. Specifically, our modality-missing-aware prompts can be plugged into multimodal transformers to handle general missing-modality cases, while only requiring less than 1% learnable parameters compared to training the entire model. We further explore the effect of different prompt configurations and analyze the robustness to missing modality. Extensive experiments are conducted to show the effectiveness of our prompt learning framework that improves the performance under various missing-modality cases, while alleviating the requirement of heavy model re-training. Code is available.
1. Introduction
The paper addresses modality missingness across samples and learning phases while adapting multimodal transformers without full-model finetuning. It proposes missing-aware prompt learning and studies prompt placement and length.
- Motivation: Multimodal transformers face modality-incomplete data in practice and increasingly costly full-model finetuning.Data completeness may be limited by privacy, device, or security constraints, while transformer models can reach billions of parameters.
- Problem Setting: The paper studies missing-modality cases that vary across samples and can occur during training, testing, or both.Examples include mixtures of text-only and image-only data in either learning phase.
- Approach: Missing-aware prompts adapt pretrained multimodal transformers to different missing-modality cases using fewer than 1% of the model’s parameters.The approach targets more affordable adaptation instead of finetuning the entire transformer.
- Prompt Designs: The study compares input-level and attention-level prompt placement in pretrained multimodal transformers.Prompt location is reported as crucial for the studied missing-modality cases.
- Prompt Configurations: Increasing the number of prompting layers improves performance, while prompt length has slight effects for attention-level prompts and stronger dataset-dependent effects for input-level prompts.The introduction reports these configuration trends without specifying a universal optimal prompt length.
- Evaluation: Extensive experiments validate the framework across various missing-modality cases while reducing the learnable parameter requirement to less than 1%.The reported contribution includes robustness analysis and prompt-configuration studies.
2. Related Work
Prior multimodal work addresses missing modalities and parameter-efficient adaptation through several model and prompt-based strategies. This paper connects prompt learning with a general missing-modality setting in multimodal transformers.
- Multimodal Transformers: Multimodal transformers fuse modalities through token concatenation and are used across diverse multimodal tasks.The related-work discussion describes them as unified models that avoid modality-specific feature extractors.
- Missing-Modality Methods: Existing missing-modality methods estimate latent features, encode missing-modality information, or predict unavailable representations from available modalities.The cited approaches include SMIL, tag encoding, and MMIN.
- Framework: The framework attaches missing-aware prompts to multimodal transformer inputs or multiple self-attention layers and uses the resulting task-token features for prediction.The described pipeline uses dummy inputs for missing modalities before prompt attachment and classification.
- Prompt Learning: Prompt learning adapts pretrained models by training a small number of learnable parameters instead of finetuning the whole model.Prior work applies prompts in language, vision, and multimodal learning for downstream adaptation and alignment.
- Positioning: The proposed framework treats different missing-modality cases as different input types and assigns corresponding missing-aware prompts.The framework is designed for cases that may vary across data samples and learning phases.
3. Proposed Method
The framework reformulates modality-incomplete multimodal inputs with dummy values and uses missing-aware prompts to adapt a frozen multimodal transformer across varying missing-modality cases. It studies input-level and attention-level prompt placement, with prompts trained alongside task-specific layers while the backbone remains frozen.
- Overall Framework: The framework supports complete, text-only, and image-only samples, including cases where missing modalities differ between training and testing.Missing modalities are represented with dummy text or image inputs to preserve the multimodal input format.
- Prompt Learning: Missing-aware prompts are assigned by missing-modality case to instruct a pretrained multimodal transformer without finetuning its backbone.The framework assigns M^2 − 1 prompts for M modality tasks and prepends them according to the input’s missing-modality type.
- Overall Framework: Only task-specific layers and prompt parameters are trainable, while the multimodal transformer parameters remain frozen.The task-specific layers include the pooler and fully connected layers, and training uses a task-specific multimodal objective.
- Input-level Prompting: Input-level prompting concatenates missing-aware prompts with the input features at selected MSA layers, allowing prompt tokens to propagate through deeper layers.With Np prompted layers and prompt length Lp, the final sequence length can become NpLp + L.
- Attention-level Prompting: Attention-level prompting splits prompts into equal-length sub-prompts prepended to the key and value vectors, leaving the output sequence length unchanged.Because prompts are not prepended to the query vector, attention-level prompting avoids the increasing sequence length of input-level prompting.
- Prompt Design: Prompts can be attached across consecutive transformer layers, and experiments identify the early half of layers starting from the first layer as the best location.The selected prompted layers are indexed continuously, with Np = N/2 in this configuration.
4. Experimental Results
Experiments evaluate missing-aware prompting across three multimodal classification datasets and varied missing-modality settings. The results show improved robustness and parameter efficiency, with input-level prompts often performing best but exhibiting greater dataset sensitivity than attention-level prompts.
- Experimental setup: Experiments cover MM-IMDb, UPMC Food-101, and Hateful Memes classification tasks under multiple modality-missing scenarios.The main quantitative comparison uses a 70% missing rate, while MM-IMDb experiments also examine wider missing-rate ranges.
- Main results: Missing-aware attention-level prompts consistently improve the baseline across the evaluated datasets and missing-modality scenarios without finetuning the entire model.Input-level prompting further improves performance in most settings, except one Hateful Memes case.
- Main results: Input-level prompting generally performs favorably across wider MM-IMDb missing-rate ranges, while attention-level prompting is more stable across datasets.The paper describes a trade-off between input-level performance and attention-level stability.
- Parameter efficiency: The method trains 221K prompt parameters, or 0.2% of ViLT, compared with 113M parameters for full finetuning, while achieving 42.66 F1-Macro versus 46.45.The full-finetuning comparison is reported for MM-IMDb under the missing-both case.
- Robustness: Input-level prompting is more robust to modality-incomplete data, whereas attention-level prompting performs better on modality-complete data.This comparison is reported as a general trend in the robustness analysis.
- Robustness: Models trained with different missing rates retain robustness across testing conditions, including cases where highly incomplete training data remains competitive at high testing missing rates.Training with more complete data performs better when testing missing rates are low.
5. Conclusions
The paper addresses missing modalities during training or testing and the heavy computational cost of transformer finetuning with missing-aware prompting. Experiments show the approach improves robustness across missing-modality scenarios without requiring heavy model finetuning.
- The method handles missing modalities that may occur differently across samples during either training or testing.
- Missing-aware prompting adapts transformer-like multimodal models without requiring heavy model finetuning.
- Extensive experiments and ablation studies demonstrate effectiveness across various missing-modality scenarios.