Source-linked AI summary
Multimodality Helps Unimodality: Cross-Modal Few-Shot Learning with Multimodal Models
Zhiqiu Lin, Samuel Yu, Zhiyi Kuang, Deepak Pathak, Deva Ramanan
TL;DR
Few-shot samples from one modality may underspecify a concept, whereas humans use cross-modal information to learn from few examples. The paper treats examples from other modalities as additional training samples in shared multimodal embedding spaces, achieving strong lightweight adaptation across visual, language, and audio settings. Its effectiveness depends on the modalities being sufficiently well aligned and pretrained.
Problem
Few-shot samples from a single modality may not sufficiently characterize a concept, leaving visual tasks underspecified.
Method
Cross-modal adaptation trains classifiers on few-shot examples from multiple modalities using encoders that map them into a shared representation space.
Results
The approach achieves state-of-the-art results with simple linear classifiers, improves prior unimodal approaches, and extends to audiovisual few-shot classification.
Takeaways & Limitations
Existing labels and cross-modal signals can serve as additional training samples, improving unimodal classifiers and supporting multimodal test applications.
Takeaways & Limitations
Cross-modal learning is less effective when modality representations are insufficiently trained or poorly aligned.
Abstract
from arXiv · showhide
The ability to quickly learn a new task with minimal instruction - known as few-shot learning - is a central aspect of intelligent agents. Classical few-shot benchmarks make use of few-shot samples from a single modality, but such samples may not be sufficient to characterize an entire concept class. In contrast, humans use cross-modal information to learn new concepts efficiently. In this work, we demonstrate that one can indeed build a better ${\bf visual}$ dog classifier by ${\bf read}$ing about dogs and ${\bf listen}$ing to them bark. To do so, we exploit the fact that recent multimodal foundation models such as CLIP learn cross-modal encoders that map different modalities to the same representation space. Specifically, we propose a simple strategy for ${\bf cross-modal}$ ${\bf adaptation}$: we treat examples from different modalities as additional few-shot examples. For example, by simply repurposing class names as an additional training sample, we trivially turn any n-shot learning problem into a (n+1)-shot problem. This allows us to produce SOTA results with embarrassingly simple linear classifiers. We show that our approach can be combined with existing methods such as prefix tuning, adapters, and classifier ensembling. Finally, to explore other modalities beyond vision and language, we construct the first (to our knowledge) audiovisual few-shot benchmark and use cross-modal training to improve the performance of both image and audio classification.
1. Introduction
Few-shot visual learning can be underspecified because limited images may not identify the intended concept. The paper proposes using multimodal examples, such as labels, audio, and images, as additional training samples for simple cross-modal adaptation.
- Few-shot visual tasks can be ambiguous when an image contains multiple plausible concepts, such as an animal, breed, hat, or background.
- Humans use multimodal signals and representations to learn concepts from few examples, motivating cross-modal adaptation of multimodal models.
- Cross-modal adaptation treats examples from different modalities as additional few-shot samples in a shared representation space.
- Using a textual label with one visual example converts an n-shot problem into an (n+1)-shot problem and supports simple linear classifiers.
- Multimodal cues can be complementary: a dog label paired with one visual example can outperform two images by clarifying the target concept.
- Unlike prior methods that use text features as classifier weights, the approach trains on textual information as additional samples and can improve prior unimodal approaches.
2. Related Works
Related work spans unimodal foundation-model adaptation, few-shot classification, multimodal pretraining, and cross-modal learning. The paper distinguishes its approach by treating other modalities as training samples rather than only as classifier weights.
- Webly supervised pretraining established large language and vision models whose learned representations transfer across downstream tasks.
- Multimodal foundation models such as CLIP and ALIGN embed images and captions into a shared representation space using contrastive learning.
- Many efficient adaptation methods borrow unimodal recipes, including linear probing, prefix tuning, and lightweight adapter networks.
- Existing CLIP adaptation methods use textual labels as classifier weights while optimizing a unimodal loss on few-shot images.
- Classic few-shot classification includes meta-learning, metric learning, transfer learning, and transductive learning, often with meta-training and episodic evaluation.
- Cross-modal learning uses additional modalities for a unimodal task without requiring instance-wise alignment, including class-level descriptions.
3. Cross-Modal Adaptation
Cross-modal adaptation extends few-shot classification by applying modality-specific encoders to shared-dimensional embeddings and training one classifier across modalities. The resulting classifier can support unimodal or multimodal inference.
- Uni-modal learning: Standard unimodal few-shot classification learns class weights from a small set of examples encoded by a pretrained feature encoder.
- Cross-modal learning: Cross-modal learning augments examples with modality labels and minimizes a loss over modality-specific encoders whose outputs share one fixed-dimensional space.
- Cross-modal learning: The shared-space requirement is satisfied by multimodal models such as CLIP and ALIGN, which map different modalities into the same embedding dimension.
- Inference: A learned classifier predicts labels for a test example from any supported modality by selecting the class with the largest weight–embedding inner product.
- Inference: The same classifier can classify different test modalities, although the paper mainly evaluates one modality to emphasize multimodality helping unimodality.
- Cross-modal ensembles: Cross-modal classifiers can be interpreted as jointly optimized ensembles of modality-specific classifiers, unlike prior methods that optimize modality-specific coefficients separately.
- Extensions: The formulation also permits multimodal test sets and future extensions using captions or attributes as additional data.
4. Vision-Language Adaptation
Vision-language adaptation treats textual class labels as additional few-shot training examples, enabling simple linear classifiers to outperform prior methods across diverse image datasets. The approach also complements prompting, adapters, and robust fine-tuning while reducing training cost.
- Cross-modal formulation: Textual class labels can be treated as explicit training samples, mapping zero-shot evaluation to one-shot cross-modal learning.The text examples and image examples need not be paired, and different numbers may be sampled per batch.
- Why cross-modal learning helps: One-shot cross-modal linear probing outperforms the 2-shot results of most prior methods, indicating complementary cues across modalities.The text and visual examples occupy somewhat different parts of the embedding space, allowing text to reshape the decision boundary.
- Vision-language results: A simple cross-modal linear probe surpasses prior methods across all shots on 11 image datasets.Partial fine-tuning of the last attentional pooling layer further improves performance, especially at higher shot counts.
- Efficiency and evaluation: The method is faster to train because features can be pre-extracted instead of relying on deep fine-tuning or heavy image augmentation.The evaluation uses fixed few-shot splits and validation-based hyperparameter tuning for fair comparison.
- Compatibility with prior methods: Cross-modal training consistently improves prompting, adapter, and robust-finetuning methods without retuning their reported hyperparameters.The reported gains suggest cross-modal adaptation is compatible with existing unimodal adaptation techniques.
5. Vision-Audio Adaptation
The paper extends cross-modal adaptation from vision-language learning to vision-audio learning with a new audiovisual benchmark. Adding one modality's examples generally improves classification in the other, although alignment and representation quality constrain the gains.
- Benchmark construction: The ImageNet-ESC benchmark intersects ImageNet objects with ESC-50 environmental sounds to support few-shot image and audio classification.It provides versions with maximal and more accurate visual-audio class matches, including cases where one sound maps to multiple visual classes.
- Audio representation: AudioCLIP supplies audio embeddings in the shared representation space used by the pretrained CLIP image and text encoders.Its audio pretraining is smaller than CLIP's, limiting its comparative strength.
- Audio-to-image transfer: Adding one audio shot improves image classification in most few-shot scenarios, including cases with loosely aligned modalities.The exception is 4-shot ImageNet-ESC-27, where noisy matches or weaker audio representations may reduce the benefit.
- Image-to-audio transfer: Adding one image shot likewise improves few-shot audio classification, supporting the approach's modality-agnostic behavior.The reverse-direction result is reported on the ImageNet-ESC benchmark.
6. Ablation Studies
Ablations show that text augmentation and test-time robustness are important aspects of cross-modal adaptation. The method remains lightweight while improving performance under distribution shifts.
- Data augmentation: Text augmentation techniques provide a sizable performance boost during vision-language adaptation.Template mining can be competitive with many carefully tuned prompts, while image-augmentation gains saturate after more than two views.
- Test-time distribution shifts: Cross-modal adaptation significantly improves robustness to ImageNet distribution shifts and achieves new state-of-the-art performance against robustness baselines.The evaluation includes ImageNet-V2, ImageNet-Sketch, ImageNet-A, and ImageNet-R.
- Efficiency: Pre-extracting features makes the approach substantially lighter and faster than methods using deep fine-tuning or heavy image augmentation.This efficiency follows from the method's lightweight training setup.
7. Discussion and Limitations
The paper presents cross-modal training as a lightweight way to adapt multimodal models for unimodal tasks, while noting that effectiveness depends on representation alignment. It reports robustness and efficiency benefits alongside broad applicability to multimodal test data.
- Cross-modal training is a lightweight approach for adapting pretrained multimodal models to downstream unimodal tasks.
- Cross-modal linear probing is more robust than its unimodal counterpart while remaining competitive with prior methods under test-time distribution shifts.
- The proposed approaches take less time and achieve state-of-the-art performance on ImageNet-16-shot.
- Cross-modal learning can address few-shot underspecification by treating text labels or other modalities as additional training samples.
- The learned models can naturally apply to multimodal test data containing both visual and audio signals.
- Cross-modal adaptation is less effective when modality representations are poorly aligned or insufficiently trained.
8. Experimental Details
The experiments use standardized CLIP feature processing, mixed image-text batches, validation-based hyperparameter tuning, and method-specific optimization settings. The appendix specifies search ranges and training procedures for the evaluated adaptation methods.
- Basic settings: CLIP features are L2-normalized before the linear layer, and text features initialize the final classifier weights.
- Basic settings: Cross-modal adaptation uses batches containing equal numbers of image and text samples.
- Basic settings: Hyperparameters are tuned on the few-shot validation set with AdamW, 50-step warmup, cosine annealing, and validation-based early stopping.
- Linear Probing: Linear probing searches learning rates [0.001, 0.0001], weight decay [0.0, 0.01, 0.0001], and batch sizes [32].
- Partial Finetuning: Partial finetuning searches learning rates [0.00001, 0.000001, 0.0000001], weight decay [0.0, 0.001, 0.00001], and uses batch size 8.
- Cross-modal Prompting: Cross-modal prompting follows CoOp with a ResNet50 backbone, 16 learnable tokens, SGD, cosine decay, and shot-dependent training epochs.
9. Additional Results
Additional experiments consistently support cross-modal adaptation across visual and audiovisual settings, including augmentation, initialization, partial finetuning, robustness, and efficiency analyses. The appendix also documents comparison and reproducibility details.
- Per-dataset results: Cross-modal adaptation consistently outperforms prior methods across a wide variety of visual recognition datasets.
- Augmentation ablation: Simple text augmentation strategies work well for visual recognition, with text augmentation having a stronger impact than image augmentation.
- Classifier initialization: Text-based classifier initialization benefits linear and partial finetuning, while cross-modal adaptation improves performance regardless of initialization.
- Partial finetuning ablation: Partial finetuning of the image encoder is more effective than finetuning the text encoder.
- ImageNet-ESC results: Including text is most performant on ImageNet-ESC, while adding text alone outperforms combining all three modalities.
- ImageNet-ESC results: In more than 75% of audiovisual experiments, adding one-shot audio or image samples outperforms unimodal linear probing.
- Comparison to ProDA: The comparison with ProDA is not direct because the methods use different dataset splits, although the proposed approach remains more performant under most tested scenarios with fewer training resources.