PaperlayerView exact revision on arXiv

Source-linked AI summary

Adaptive Cross-Modal Few-Shot Learning

Chen Xing, Negar Rostamzadeh, Boris N. Oreshkin, Pedro O. Pinheiro

arXiv:1902.07104v3cs.LGstat.ML

TL;DR

Few-shot classification is difficult with limited visual support, while visual and semantic representations can contain complementary information. The paper proposes AM3, which adaptively combines the two modalities, and reports large gains over unimodal and alignment methods, especially in one-shot settings.

  • Problem

    Few-shot classification must learn new image categories from limited visual support, although visual and semantic spaces have different structures and may provide unequal information.

  • Method

    AM3 adaptively combines visual and semantic representations through a category-dependent convex mixture within metric-based meta-learning.

  • Results

    AM3 outperforms metric-based backbones, unimodal methods, and modality-alignment methods across tested datasets and settings, with particularly large gains in one-shot scenarios.

  • Takeaways & Limitations

    Adaptive use of semantic information is especially helpful when the visual modality provides very little information.

  • Takeaways & Limitations

    The paper’s motivating hypothesis specifically concerns scenarios such as visually similar objects or very small numbers of visual shots.

Abstract

from arXiv · show

Metric-based meta-learning techniques have successfully been applied to few-shot classification problems. In this paper, we propose to leverage cross-modal information to enhance metric-based few-shot learning methods. Visual and semantic feature spaces have different structures by definition. For certain concepts, visual features might be richer and more discriminative than text ones. While for others, the inverse might be true. Moreover, when the support from visual information is limited in image classification, semantic representations (learned from unsupervised text corpora) can provide strong prior knowledge and context to help learning. Based on these two intuitions, we propose a mechanism that can adaptively combine information from both modalities according to new image categories to be learned. Through a series of experiments, we show that by this adaptive combination of the two modalities, our model outperforms current uni-modality few-shot learning methods and modality-alignment methods by a large margin on all benchmarks and few-shot scenarios tested. Experiments also show that our model can effectively adjust its focus on the two modalities. The improvement in performance is particularly large when the number of shots is very small.

1 Introduction

Few-shot learning addresses learning new concepts from very limited labeled data, where unimodal visual methods can struggle. AM3 instead combines visual and semantic information adaptively according to category and scenario.

  • Motivation: Few-shot learning targets new-concept recognition from very few labeled examples, addressing overfitting and generalization problems caused by limited data.Human visual learning provides motivation for leveraging prior knowledge and context in this regime.
  • Motivation: Semantic features from text can provide useful information when visual data is limited, complementing recent unimodal progress in few-shot image classification.
  • Motivation: Visual and semantic feature spaces are heterogeneous: either modality may offer richer or more discriminative information for a given concept.With very few support images, visual information may be noisy and local, whereas text-derived representations can provide general prior knowledge and context.
  • Method: AM3 adaptively combines visual and semantic representations instead of forcing them into an aligned space during few-shot classification.Its adaptive convex combination shifts toward semantic context or local visual detail according to the category.
  • Contributions: AM3 is proposed for cross-modal few-shot classification and is reported to improve metric-based methods, outperform state-of-the-art approaches, and adjust modality focus quantitatively.The contributions cover different datasets and numbers of shots, including unimodal and cross-modal comparisons.

2 Related Work

Few-shot learning builds on meta-learning methods, including metric-based and gradient-based approaches, but differs from zero-shot learning because test-time support images are available. The paper argues for preserving modality-specific structures and combining them adaptively rather than directly applying zero-shot alignment.

  • Few-shot learning: Few-shot meta-learning methods are broadly divided into metric-based approaches and gradient-based approaches.
  • Few-shot learning: Metric-based methods learn representations that bring same-class samples closer and separate different classes within episodic few-sample training tasks.Matching networks and prototypical networks exemplify nearest-neighbor and prototype-based metric learning.
  • Few-shot learning: Gradient-based methods seek models that generalize to new tasks after only a few fine-tuning updates, with many approaches built on MAML.
  • Cross-modal extension: Existing metric-based few-shot approaches rely solely on visual features, while the paper’s contribution is designed to integrate orthogonally with them.
  • Zero-shot learning: Zero-shot methods generally align visual and auxiliary modalities because they lack visual information for unseen categories, whereas few-shot learning has support images at test time.The paper therefore presents alignment as unnecessarily rigid for few-shot classification and favors preserving both modality structures.

3 Method

AM3 extends metric-based few-shot classification by combining visual prototypes with semantic label information, adapting the balance between modalities for each category. It retains episodic prototypical-network training while comparing queries against cross-modal prototypes.

  • 3.1 Episodic Training: Few-shot models train on episodic K-shot, N-way tasks with support examples and separate query examples from the same sampled categories.Training minimizes query prediction loss conditioned on the episode’s support set.
  • 3.1.2 Prototypical Networks: Prototypical networks classify queries by comparing their learned visual embeddings with category prototypes computed from support-set embeddings.Each prototype is the average embedding of the support samples belonging to its category, and class probabilities use a softmax over negative distances.
  • 3.2 Adaptive Modality Mixture Mechanism: Semantic features can distinguish concepts that visual features find difficult, while visual features can be richer for other concepts because the modalities have heterogeneous structures.Examples include Siberian husky versus wolf and komondor versus mop.
  • 3.2 Adaptive Modality Mixture Mechanism: The adaptive coefficient lets AM3 adjust its focus between visual and semantic information according to image categories and the available visual support.Semantic information is especially useful when support images are very limited or noisy, whereas visual information may be more discriminative for some concepts.
  • 3.2 Adaptive Modality Mixture Mechanism: AM3 transforms each category’s word embedding into the visual prototype space and forms a convex combination of the visual prototype and transformed semantic representation.The mixture coefficient is category-conditioned, allowing the method to combine the two modalities rather than align them into one shared structure.
  • 3.2 Adaptive Modality Mixture Mechanism: AM3 replaces visual-only prototype distances with distances between queries and cross-modal prototypes while retaining a prototypical-network-like training procedure.The method can potentially be applied to metric-based approaches that calculate category prototypes, and is applied to ProtoNets and TADAM.

4 Experiments

Experiments evaluate AM3 against unimodal, modality-alignment, and metric-based extensions across few-shot datasets and shot settings. AM3 consistently improves over its backbones and competing methods, with especially large gains when visual support is scarce.

  • Experimental Setup: Experiments compare AM3 with unimodal few-shot, modality-alignment, and metric-based modality-alignment baselines on miniImageNet, tieredImageNet, and CUB-200.AM3 is tested with ProtoNets++ and TADAM backbones, while CUB-200 enables comparison with published modality-alignment results.
  • Results: AM3 outperforms its backbone methods by a large margin in all tested cases on miniImageNet and tieredImageNet.The results indicate that text information can boost metric-based few-shot learning when properly incorporated.
  • Results: AM3 with TADAM outperforms current unimodal and modality-alignment state-of-the-art methods, particularly in the 1-shot scenario.Its margin over both baseline families increases as the number of shots decreases.
  • Results: Modality-alignment baselines improve when extended to metric-based episodic learning, but most remain below the current state-of-the-art unimodal few-shot method.The authors suggest that forcing distinct visual and semantic structures to align may discard information in few-shot learning.
  • Results: On CUB-200, the conclusions above also hold, and additional zero-shot and generalized few-shot experiments assess the proposed adaptive mechanism.These experiments provide a further comparison with modality-alignment methods and test the mechanism beyond the main few-shot benchmarks.
  • Adaptiveness Analysis: As shot count increases, the performance gap between AM3 and its backbones decreases, while the mean mixing coefficient λ shifts toward less reliance on text.The variance of λ also decreases with shot count and tracks the shrinking performance gap, supporting category-level adaptiveness.

5 Conclusion

AM3 adaptively leverages cross-modal information for few-shot classification and improves metric-based approaches across datasets and settings. Its semantic features are especially helpful with very little visual data, while quantitative analysis shows adaptive focus across modalities.

  • Conclusion: AM3 adaptively leverages cross-modal information and boosts metric-based few-shot classification across different datasets and settings.The method uses unsupervised textual data and is reported to outperform state-of-the-art few-shot classification methods by a large margin.
  • Conclusion: Semantic features are particularly helpful in the very low visual-data regime, including one-shot classification.Quantitative experiments also show that AM3 can adjust its focus between the two modalities.

A Algorithm for Episode Loss

Algorithm 1 samples episodic classes and computes cross-modal prototypes before accumulating the episode loss over query examples.

  • Episode construction: Each training episode samples N classes from the M-class training set.The episode uses K supports and KQ queries per class.
  • Cross-modal prototypes: For each selected class, the algorithm looks up its label embedding and transforms it into the semantic representation used for prototype construction.
  • Cross-modal prototypes: The class prototype is formed by mixing visual and semantic representations with an adaptive coefficient λc.The mixture combines pc and wc as λc · pc + (1 − λc) · wc.
  • Loss computation: The algorithm then computes the episodic loss over query examples from each selected class.

B Descriptions of data sets

The experiments use ImageNet-derived datasets with different scales and splits, plus fine-grained CUB-200 attributes and pretrained word embeddings as semantic inputs.

  • miniImageNet: miniImageNet contains 100 categories with 600 84 × 84 images each, split into 64 training, 16 validation, and 20 test categories.
  • tieredImageNet: tieredImageNet contains 34 high-level nodes and 779,165 images, yielding 351 training, 97 validation, and 160 test categories.Its splits follow the higher-level ImageNet label hierarchy.
  • CUB-200: CUB-200 provides 11,788 images from 200 bird types annotated with 312 attributes used as the semantic modality.
  • Word embeddings: Category-label semantics are represented with 300-dimensional GloVe embeddings trained on 840B Common Crawl tokens.Multiword annotations are averaged, and fastText produced similar performances in an additional experiment.

C Baselines

The baselines include modality-alignment methods evaluated under few-shot settings, with test-class prototypes computed in method-specific representation spaces.

  • Modality-alignment baselines: CADA-VAE-style alignment baselines add randomly sampled N-shot test-class images to training and compare image queries with class embeddings in aligned space.This evaluation follows zero-shot and generalized zero-shot decision procedures during testing.
  • Meta-learning extensions: Meta-learning extensions train on standard episodes from training classes and add an auxiliary loss to match the few-shot visual test scenario.
  • Prototype computation: Prototype representations are computed in each method’s designated space, including visual, latent, or discriminator-input space.DeViSE uses its pre-transformer visual space; ReViSE and CADA-VAE use latent space; f-CLSWGAN uses discriminator input space.

D Implementation Details of AM3 Experiments

AM3 experiments use a ResNet-12 visual backbone, neural semantic and mixing transformations, stochastic-gradient training, and repeated episodic evaluation with confidence intervals.

  • Model architecture: The ResNet-12 visual extractor produces 512-dimensional embeddings and is used for AM3 and modality-alignment baselines.The same backbone also defines the ProtoNets++ implementation.
  • Model architecture: The semantic transformation outputs 512 dimensions, while the mixture transformation outputs one scalar λc for each class.Both networks have one hidden layer with 300 units, ReLU nonlinearities, and dropout.
  • Optimization: Models are trained with momentum SGD using an initial learning rate of 0.1 and momentum coefficient 0.9.Training schedules differ between miniImageNet and tieredImageNet.
  • Episodic evaluation: Training batches combine multiple five-class tasks, with task and query counts selected separately for 1-shot, 5-shot, and 10-shot settings.Evaluation samples 1,000 test tasks, each containing 100 random queries.
  • Reporting: Results average five independent runs with 95% confidence intervals, while hyperparameters are selected using validation accuracy.

E Results on CUB-200

On CUB-200, the experiments compare AM3 with modality-alignment and generalized few-shot baselines. AM3-ProtoNets outperforms CADA-VAE in every generalized few-shot case tested, while the adaptive mechanism accounts for the boost over DeViSE.

  • Comparison with modality-alignment baselines: Without its adaptive mechanism, AM3 performs roughly the same as DeViSE in the 0-shot scenario.In this setting, AM3 reduces to a modality-alignment method mapping the text semantic space to the visual space.
  • Few-shot classification: Results on the other CUB-200 few-shot cases are consistent with those on the paper’s two other few-shot learning datasets.
  • Generalized few-shot learning: AM3-ProtoNets outperforms CADA-VAE in every generalized few-shot case tested on CUB-200.Performance is evaluated using H-acc, the harmonic mean of seen- and unseen-class accuracy.

F Ablation study on the input of the adaptive mechanism

The ablation varies the inputs to AM3’s adaptive mixing network across two datasets. Transformed GloVe and semantic conditioning perform better than their alternatives, with query-semantic conditioning depending on the backbone.

  • Ablation inputs: The ablation tests raw GloVe embeddings, visual representations, and concatenated query-language embeddings as inputs to the adaptive mixing network.The table also includes the original model as its last row.
  • Feature transformations: Conditioning on transformed GloVe features performs better than conditioning on raw features.
  • Modality conditioning: Semantic-feature conditioning performs better than visual-feature conditioning, suggesting semantic space is more appropriate for the adaptive mechanism.
  • Backbone dependence: Query-semantic conditioning helps with the ProtoNets++ backbone but not with TADAM.
Loading 1902.07104v3…