PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-level Semantic Feature Augmentation for One-shot Learning

Zitian Chen, Yanwei Fu, Yinda Zhang, Yu-Gang Jiang, Xiangyang Xue, Leonid Sigal

arXiv:1804.05298v4cs.CV

TL;DR

One-shot learning lacks sufficient labeled data, motivating augmentation that can preserve semantic relationships. The paper proposes Dual TriNet to map multi-layer visual features into semantic space, augment them, and decode new features. Experiments report improved few-shot classification across four datasets, with complementary augmentation strategies and layers improving results further.

  • Problem

    Few-shot learning must recognize novel categories from very few examples, while insufficient training data is the main obstacle to learning good classifiers.

  • Method

    Dual TriNet encodes multi-layer ResNet-18 visual features into semantic space, augments or replaces related semantic representations, and decodes them into synthetic feature instances.

  • Results

    The framework improves few-shot classification across four datasets, with combined semantic augmentation strategies and multi-layer features producing the strongest reported results.

  • Takeaways & Limitations

    Semantic-space augmentation yields complex non-Gaussian feature distributions and provides complementary synthesized features that improve one-shot classification.

Abstract

from arXiv · show

The ability to quickly recognize and learn new visual concepts from limited samples enables humans to swiftly adapt to new environments. This ability is enabled by semantic associations of novel concepts with those that have already been learned and stored in memory. Computers can start to ascertain similar abilities by utilizing a semantic concept space. A concept space is a high-dimensional semantic space in which similar abstract concepts appear close and dissimilar ones far apart. In this paper, we propose a novel approach to one-shot learning that builds on this idea. Our approach learns to map a novel sample instance to a concept, relates that concept to the existing ones in the concept space and generates new instances, by interpolating among the concepts, to help learning. Instead of synthesizing new image instance, we propose to directly synthesize instance features by leveraging semantics using a novel auto-encoder network we call dual TriNet. The encoder part of the TriNet learns to map multi-layer visual features of deep CNNs, that is, multi-level concepts, to a semantic vector. In semantic space, we search for related concepts, which are then projected back into the image feature spaces by the decoder portion of the TriNet. Two strategies in the semantic space are explored. Notably, this seemingly simple strategy results in complex augmented feature distributions in the image feature space, leading to substantially better performance.

I. INTRODUCTION

Few-shot learning addresses recognition when labeled examples are scarce, but existing augmentation can introduce semantic noise or require domain expertise. The paper therefore motivates semantic-space feature augmentation as a way to generate informative, varied representations.

  • Few-shot learning must build classifiers from very few or single examples because obtaining many labeled instances can be expensive or infeasible.
  • Existing augmentation may borrow related-category data, but semantic noise can cause negative transfer and reduce performance.
  • Synthetic images can be semantically correct but may require careful domain adaptation to transfer knowledge and features.
  • Feature augmentation has been relatively little explored, and prior attribute-guided synthesis relies on predefined semantic attributes.
  • Random noise may not substantially inform the decision boundary, whereas semantic spaces place similar concepts near one another and provide semantic continuity for augmentation.
  • Dual TriNet maps multi-layer visual features into semantic space, perturbs or replaces them with related semantic representations, and decodes augmented features for classification.

C. Embedding Network structures

The paper frames one-shot learning around base classes with sufficient data and novel classes evaluated from a small support set. Its visual-semantic embedding setting allows class semantics to be represented through attributes, word vectors, or learned relationships.

  • Word-vector semantics are obtained from vocabulary entities learned by word2vec on a large-scale corpus.
  • Visual-semantic embedding methods project visual features and semantic entities into a common space, while this work directly augments instance features.
  • One-shot learning separates base categories with adequate labeled data from novel categories evaluated using only a small support set.
  • Semantic vectors may represent classes through human-defined attributes, word vectors, or a subspace learned from semantic class relationships.
  • The test set contains novel classes and is unavailable during training, while training uses base data and the novel support set.

B. Overview

The framework extracts multi-layer image features with ResNet-18 and uses Dual TriNet to synthesize additional feature instances. These real and synthesized features jointly train the one-shot classifier through an end-to-end architecture.

  • The method directly augments training-instance features and uses synthesized features as additional training instances for one-shot learning.
  • The ResNet-18 and Dual TriNet architecture is trained end-to-end by combining the loss functions of both networks.
  • ResNet-18 converts raw images into feature maps from four sequential residual layers, providing a multi-level image-feature encoding.
  • Dual TriNet uses an encoder to map visual features into semantic space and a decoder to project augmented semantic representations back into feature space.

C. Dual TriNet Network

Dual TriNet learns a shared mapping between hierarchical ResNet-18 features and semantic representations, enabling augmentation in semantic space before decoding. The architecture exploits complementary information across layers and supports multiple semantic-space constructions.

  • The framework extracts ResNet-18 features, encodes them into semantic space, augments them there, and decodes the results into feature space for classifier training.
  • Dual TriNet addresses the difficulty of mapping hierarchical features from different CNN layers into semantic space.
  • The semantic space can be a human-defined attribute space, a word-vector space, or a subspace derived from class relationships.
  • SVD of the cosine-similarity relationship matrix produces a unitary semantic space whose rows represent class semantic vectors.
  • The encoder TriNet combines features from all ResNet-18 layers to map an instance toward its semantic vector, exploiting differences and complementary information.
  • The decoder is trained on base data and synthesizes novel-class instances as perturbations of layer-specific feature representations.

D. Feature Augmentation by Dual TriNet

Dual TriNet augments few-shot features by encoding multi-layer visual representations into semantic space, perturbing or replacing them with semantic neighbors, and decoding them back into feature space.

  • The augmentation is applied only to extend Dsupport using features generated by the learned dual TriNet.
  • Semantic Gaussian (SG): Semantic Gaussian augmentation samples vectors after mapping multi-layer ResNet-18 features into semantic space, with σ controlling noise magnitude.
  • Semantic Neighborhood (SN): Semantic neighborhood augmentation samples vectors near the encoded instance from vocabulary learned by word2vec, then decodes those neighbors into synthesized features.
  • The encoder uses the instance representation rather than its class word vector because the latter represents only the class semantic center.
  • 3 ∼5% performance drop occurs on average in 1-shot/5-shot classification when augmenting from the class word vector instead of the encoded instance representation.
  • Directly adding Gaussian noise to multi-layer features does not significantly improve one-shot classification on miniImageNet, unlike semantic-space augmentation.

E. One-shot Classification

For novel instances, the framework encodes features from all layers, generates semantic-space augmentations, decodes them into feature representations, and trains a supervised one-shot classifier.

  • Novel instances are mapped from all ResNet-18 layer features into semantic vectors using the encoder TriNet.
  • Semantic Gaussian and neighborhood operations generate multiple synthetic instances in addition to each original training instance.
  • The decoder TriNet maps each new semantic vector back into features from all network layers.
  • Non-final-layer synthesized features are propagated through later feature-extractor layers to obtain final-layer features for classification.
  • Original and synthesized final-layer features train the one-shot classifier, with all augmented vectors assigned the original instance’s class label.
  • KNN, SVM, and Logistic Regression are used as supervised one-shot classifiers for test-sample final-layer features.

A. Datasets

Experiments use four image datasets with fixed train, validation, and test class splits, while keeping network and input settings consistent across datasets.

  • All datasets use ResNet-18 trained only on the training split, with identical networks, parameters, and 224 × 224 input images.
  • miniImageNet contains 60,000 images from 100 classes, split into 64 training, 16 validation, and 20 testing classes.
  • Cifar-100 contains 60,000 images across 100 fine-grained and 20 coarse-level categories, using 64/16/20 train-validation-test class splits.
  • CUB-200 contains 11,788 bird images from 200 categories, split into 100 training, 50 validation, and 50 testing classes, with 312-dimensional class attributes.
  • Caltech-256 contains 30,607 images from 256 classes, split into 150 training, 56 validation, and 50 testing classes.

B. Network structures and Settings

The experiments use a shared ResNet-18 and dual TriNet setup across datasets, semantic spaces, classifiers, and augmentation configurations. Results show that semantic feature augmentation improves one-shot learning, with multi-layer and combined semantic augmentations providing complementary benefits.

  • Settings: The same ResNet-18 and dual TriNet are used across all four datasets and experiments.
  • Settings: Each training instance produces four semantic augmentations per layer, yielding 16 synthesized fourth-layer instances and 17 total training instances.
  • Settings: Experiments evaluate KNN, SVM, and LR classifiers, with classifier hyperparameters selected by validation-set cross-validation.
  • Experimental results: Multi-layer augmentation outperforms individual layer augmentation on CUB-200 and miniImageNet, indicating complementary information across layers.
  • Experimental results: Augmented features improve one-shot performance across KNN, SVM, and LR classifiers.
  • Experimental results: On CUB-200, SG, SN, and AG each outperform no augmentation; combining them further improves accuracy, with all three achieving the highest one-shot result.

E. Experimental results on Caltech-256 and CIFAR-100

On Caltech-256 and CIFAR-100, the proposed framework uses semantic word-vector augmentation across ResNet-18 layers and outperforms competing methods. Standard image and Gaussian feature augmentations do not improve one-shot accuracy.

  • Experimental setup: The method synthesizes 16 augmented features per semantic strategy from all four ResNet-18 layers and uses an SVM classifier.The semantic strategies are SG and SN in the semantic word-vector space.
  • Results: The augmented instance features provide a clear improvement over the ResNet-18 baseline on both datasets.ResNet-18 remains a strong baseline, surpassed mainly by the method using ResNet-50 in DEML+Meta-SGD.
  • Comparison with standard augmentation: Cropping, rotation, flipping, color transformations, and Gaussian noise added to ResNet-18 features fail to improve one-shot classification accuracy.These standard augmentation methods are evaluated against the proposed feature augmentation approach.

B. Dual TriNet structure

Dual TriNet uses an encoder-decoder architecture to perturb multi-layer visual features through semantic space and reconstruct augmented visual features. This preserves visual information while adding semantic variation for one-shot classification.

  • Dual TriNet structure: Dual TriNet learns transformations between multi-layer ResNet-18 image features and a semantic space using encoder and decoder TriNets.The encoder maps extracted features into semantic space, while the decoder maps perturbed semantic features back to multi-layer visual feature space.
  • Ablation: Using the true-label semantic vector instead of the encoder output causes augmented samples to hurt performance.The result supports using encoded instance information rather than replacing it with only the label representation.
  • Ablation: Classifying directly in semantic space reduces performance by over 5% because visual information is lost during mapping.The proposed decoder retains original information while incorporating additional semantic information.
  • Conclusion: The framework directly augments multi-layer visual features and improves few-shot classification across four datasets.The study focuses mainly on classification and notes possible extension to segmentation and other feature extractors.
Loading 1804.05298v4…