Multi-level Semantic Feature Augmentation for One-shot Learning

• Few-shot learning must build classifiers from very few or single examples because obtaining many labeled instances can be expensive or infeasible.Source: §I · p. 1
• Existing augmentation may borrow related-category data, but semantic noise can cause negative transfer and reduce performance.Source: §I · p. 1
• Synthetic images can be semantically correct but may require careful domain adaptation to transfer knowledge and features.Source: §I · p. 1
• Feature augmentation has been relatively little explored, and prior attribute-guided synthesis relies on predefined semantic attributes.Source: §I · p. 1
• Random noise may not substantially inform the decision boundary, whereas semantic spaces place similar concepts near one another and provide semantic continuity for augmentation.Source: §I · p. 1
• Dual TriNet maps multi-layer visual features into semantic space, perturbs or replaces them with related semantic representations, and decodes augmented features for classification.Source: §I · pp. 1–2

• Word-vector semantics are obtained from vocabulary entities learned by word2vec on a large-scale corpus.Source: C. Embedding Network structures · p. 3
• Visual-semantic embedding methods project visual features and semantic entities into a common space, while this work directly augments instance features.Source: C. Embedding Network structures · p. 3
• One-shot learning separates base categories with adequate labeled data from novel categories evaluated using only a small support set.Source: A. Problem setup · p. 3
• Semantic vectors may represent classes through human-defined attributes, word vectors, or a subspace learned from semantic class relationships.Source: A. Problem setup · p. 3
• The test set contains novel classes and is unavailable during training, while training uses base data and the novel support set.Source: A. Problem setup · p. 3

• The method directly augments training-instance features and uses synthesized features as additional training instances for one-shot learning.Source: B. Overview · p. 3
• The ResNet-18 and Dual TriNet architecture is trained end-to-end by combining the loss functions of both networks.Source: B. Overview · p. 3
• ResNet-18 converts raw images into feature maps from four sequential residual layers, providing a multi-level image-feature encoding.Source: B. Overview · p. 3
• Dual TriNet uses an encoder to map visual features into semantic space and a decoder to project augmented semantic representations back into feature space.Source: B. Overview · p. 3

• The framework extracts ResNet-18 features, encodes them into semantic space, augments them there, and decodes the results into feature space for classifier training.Source: Figure 1 · p. 4
• Dual TriNet addresses the difficulty of mapping hierarchical features from different CNN layers into semantic space.Source: C. Dual TriNet Network · pp. 3–4
• The semantic space can be a human-defined attribute space, a word-vector space, or a subspace derived from class relationships.Source: C. Dual TriNet Network · p. 4
• SVD of the cosine-similarity relationship matrix produces a unitary semantic space whose rows represent class semantic vectors.Source: C. Dual TriNet Network · p. 4
• The encoder TriNet combines features from all ResNet-18 layers to map an instance toward its semantic vector, exploiting differences and complementary information.Source: C. Dual TriNet Network · p. 4
• The decoder is trained on base data and synthesizes novel-class instances as perturbations of layer-specific feature representations.Source: C. Dual TriNet Network · p. 4

• The augmentation is applied only to extend Dsupport using features generated by the learned dual TriNet.Source: D. Feature Augmentation by Dual TriNet · p. 4

Semantic Gaussian (SG)

• Semantic Gaussian augmentation samples vectors after mapping multi-layer ResNet-18 features into semantic space, with σ controlling noise magnitude.Source: D. Feature Augmentation by Dual TriNet · p. 4

Semantic Neighborhood (SN)

• Semantic neighborhood augmentation samples vectors near the encoded instance from vocabulary learned by word2vec, then decodes those neighbors into synthesized features.Source: D. Feature Augmentation by Dual TriNet · p. 5

• The encoder uses the instance representation rather than its class word vector because the latter represents only the class semantic center.Source: D. Feature Augmentation by Dual TriNet · p. 5
• 3 ∼5% performance drop occurs on average in 1-shot/5-shot classification when augmenting from the class word vector instead of the encoded instance representation.Source: D. Feature Augmentation by Dual TriNet · p. 5
• Directly adding Gaussian noise to multi-layer features does not significantly improve one-shot classification on miniImageNet, unlike semantic-space augmentation.Source: D. Feature Augmentation by Dual TriNet · p. 5

• Novel instances are mapped from all ResNet-18 layer features into semantic vectors using the encoder TriNet.Source: E. One-shot Classification · p. 5
• Semantic Gaussian and neighborhood operations generate multiple synthetic instances in addition to each original training instance.Source: E. One-shot Classification · p. 5
• The decoder TriNet maps each new semantic vector back into features from all network layers.Source: E. One-shot Classification · p. 5
• Non-final-layer synthesized features are propagated through later feature-extractor layers to obtain final-layer features for classification.Source: E. One-shot Classification · p. 5
• Original and synthesized final-layer features train the one-shot classifier, with all augmented vectors assigned the original instance’s class label.Source: E. One-shot Classification · p. 5
• KNN, SVM, and Logistic Regression are used as supervised one-shot classifiers for test-sample final-layer features.Source: E. One-shot Classification · p. 5

• All datasets use ResNet-18 trained only on the training split, with identical networks, parameters, and 224 × 224 input images.Source: A. Datasets · p. 5
• miniImageNet contains 60,000 images from 100 classes, split into 64 training, 16 validation, and 20 testing classes.Source: A. Datasets · p. 5
• Cifar-100 contains 60,000 images across 100 fine-grained and 20 coarse-level categories, using 64/16/20 train-validation-test class splits.Source: A. Datasets · p. 5
• CUB-200 contains 11,788 bird images from 200 categories, split into 100 training, 50 validation, and 50 testing classes, with 312-dimensional class attributes.Source: A. Datasets · p. 5
• Caltech-256 contains 30,607 images from 256 classes, split into 150 training, 56 validation, and 50 testing classes.Source: A. Datasets · p. 5

Settings

• The same ResNet-18 and dual TriNet are used across all four datasets and experiments.Source: B. Network structures and Settings · p. 5
• Each training instance produces four semantic augmentations per layer, yielding 16 synthesized fourth-layer instances and 17 total training instances.Source: B. Network structures and Settings · p. 6
• Experiments evaluate KNN, SVM, and LR classifiers, with classifier hyperparameters selected by validation-set cross-validation.Source: C. Competitors and Classification models · p. 6

Experimental results

• Multi-layer augmentation outperforms individual layer augmentation on CUB-200 and miniImageNet, indicating complementary information across layers.Source: D. Experimental results on… · p. 7, Figure 2 · p. 7
• Augmented features improve one-shot performance across KNN, SVM, and LR classifiers.Source: D. Experimental results on… · p. 7, Figure 2 · p. 7
• On CUB-200, SG, SN, and AG each outperform no augmentation; combining them further improves accuracy, with all three achieving the highest one-shot result.Source: D. Experimental results on… · pp. 7–8, Figure 3 · p. 8

Experimental setup

• The method synthesizes 16 augmented features per semantic strategy from all four ResNet-18 layers and uses an SVM classifier.The semantic strategies are SG and SN in the semantic word-vector space.Source: E. Experimental results on Caltech-256… · p. 8

Results

• The augmented instance features provide a clear improvement over the ResNet-18 baseline on both datasets.ResNet-18 remains a strong baseline, surpassed mainly by the method using ResNet-50 in DEML+Meta-SGD.Source: E. Experimental results on Caltech-256… · p. 8

Comparison with standard augmentation

• Cropping, rotation, flipping, color transformations, and Gaussian noise added to ResNet-18 features fail to improve one-shot classification accuracy.These standard augmentation methods are evaluated against the proposed feature augmentation approach.Source: A. Comparison with standard… · p. 8

Dual TriNet structure

• Dual TriNet learns transformations between multi-layer ResNet-18 image features and a semantic space using encoder and decoder TriNets.The encoder maps extracted features into semantic space, while the decoder maps perturbed semantic features back to multi-layer visual feature space.Source: B. Dual TriNet structure · p. 10

Ablation

• Using the true-label semantic vector instead of the encoder output causes augmented samples to hurt performance.The result supports using encoded instance information rather than replacing it with only the label representation.Source: B. Dual TriNet structure · p. 10
• Classifying directly in semantic space reduces performance by over 5% because visual information is lost during mapping.The proposed decoder retains original information while incorporating additional semantic information.Source: B. Dual TriNet structure · p. 10

Conclusion

• The framework directly augments multi-layer visual features and improves few-shot classification across four datasets.The study focuses mainly on classification and notes possible extension to segmentation and other feature extractors.Source: §VI · p. 10