Source-linked AI summary
Few-Shot Learning via Saliency-guided Hallucination of Samples
Hongguang Zhang, Jing Zhang, Piotr Koniusz
TL;DR
Few-shot learning requires models to learn new concepts from very limited data, while many existing hallucination generators require substantial annotated training data. The paper introduces saliency-guided feature-space hallucination using foreground-background mixing, and reports state-of-the-art results on two public few-shot datasets.
Problem
Few-shot learning must support learning new concepts from limited samples, while specialized datapoint generators can themselves require large numbers of annotated training examples.
Method
SalNet uses saliency maps to obtain foregrounds and backgrounds, mixes them in feature space through a two-stream network, and applies regularization and similarity-based mixing strategies.
Results
The authors report state-of-the-art performance on two challenging publicly available few-shot learning datasets.
Takeaways & Limitations
Saliency maps can support hallucinating additional feature-space datapoints for few-shot learning without the paper’s described costly hallucination setup.
Takeaways & Limitations
Performance depends somewhat on the saliency detector, and complex scenes can produce substandard maps or incompatible foreground-background pairs.
Abstract
from arXiv · showhide
Learning new concepts from a few of samples is a standard challenge in computer vision. The main directions to improve the learning ability of few-shot training models include (i) a robust similarity learning and (ii) generating or hallucinating additional data from the limited existing samples. In this paper, we follow the latter direction and present a novel data hallucination model. Currently, most datapoint generators contain a specialized network (i.e., GAN) tasked with hallucinating new datapoints, thus requiring large numbers of annotated data for their training in the first place. In this paper, we propose a novel less-costly hallucination method for few-shot learning which utilizes saliency maps. To this end, we employ a saliency network to obtain the foregrounds and backgrounds of available image samples and feed the resulting maps into a two-stream network to hallucinate datapoints directly in the feature space from viable foreground-background combinations. To the best of our knowledge, we are the first to leverage saliency maps for such a task and we demonstrate their usefulness in hallucinating additional datapoints for few-shot learning. Our proposed network achieves the state of the art on publicly available datasets.
1. Introduction
The paper proposes saliency-guided feature-space data hallucination for few-shot learning, combining foregrounds and backgrounds to generate additional samples. It introduces mixing strategies and regularization, and reports state-of-the-art results on two public datasets.
- The work addresses few-shot learning, where CNN-based systems require many labeled examples while humans can learn novel concepts from few samples.
- SalNet uses a pre-trained saliency network to separate foregrounds and backgrounds, then mixes them in an encoder’s feature space to hallucinate samples.
- The method regularizes mixed representations using Real Representation Regularization and selects backgrounds through intra-class or inter-class mixing strategies.
- The authors propose a novel saliency-guided data hallucination network for few-shot learning.
- The paper investigates hallucination strategies, a regularization method, and two strategies intended to prevent substandard hallucinated samples.
- The authors report that SalNet is the first method, to their knowledge, to use saliency maps for datapoint hallucination and achieves state-of-the-art performance on two public datasets.
2. Related Work
Related work covers few-shot learning, hallucination-based augmentation, saliency detection, and second-order statistics. The paper positions its approach as decomposing images into saliency-guided foreground and background representations for feature-space mixing.
- 2.1. Learning From Few Samples: Few-shot methods include generative transfer, metric-learning networks, support-set protocols, class prototypes, meta-learning, relation learning, and second-order similarity descriptors.
- 2.1. Learning From Few Samples: Hallucination approaches generate plausible combinations from manually clustered descriptors, whereas Mixup forms convex combinations of datapoints and labels.
- 2.2. Saliency Detection: The paper instead decomposes images into foreground and background representations using saliency maps, then mixes these pairs to hallucinate auxiliary training samples.
- 2.2. Saliency Detection: Saliency detectors produce per-pixel relevance maps for foreground objects, but conventional detectors can underperform on complex scenes and deep models require pixel-wise labels.
- 2.2. Saliency Detection: The proposed pipeline uses Saliency Net, FEMN, and Similarity Net, with FEMN combining foreground-background streams before second-order feature aggregation.
- 2.2. Saliency Detection: The method compares a weakly supervised deep saliency detector with fully supervised RFCN and unsupervised RBD alternatives.
- 2.3. Second-order Statistics: Second-order statistics are established in texture recognition, semantic segmentation, and object category recognition, with power normalization used to limit burstiness.
3. Approach
SalNet uses saliency maps to separate foregrounds and backgrounds, then hallucinates feature-space samples through foreground-background mixing and similarity learning. The approach addresses scarce few-shot training data but depends on saliency quality and pair compatibility.
- Network: The pipeline extends Relation Net by encoding images, forming query-support episodes, and learning whether pairs share a class.The similarity network is trained with mean squared error over episode query-support pairs.
- Network: SalNet generates foreground hypotheses, FEMN mixes foreground-background representations, and SimNet learns query-support similarity.Foregrounds and backgrounds are isolated with saliency maps before feature-space mixing.
- Saliency Map Generation: Saliency detectors provide the foreground-background prior, with MNL used primarily and RFCN and RBD evaluated as alternatives.RBD is unsupervised and inexpensive, whereas MNL and RFCN are deep saliency approaches.
- Saliency Map Generation: Complex scenes expose detector sensitivity: the unsupervised method can miss foregrounds, and RBD-based few-shot performance is expected to be worse.The dataset includes both simple and complex scenes, making detector choice consequential for the method.
- Data Hallucination: Intra-class hallucination mixes each foreground only with same-class backgrounds, producing W −1 new datapoints per image but not working in one-shot settings.Its simple summation-and-refinement strategy can introduce noise from poor saliency maps or incompatible combinations.
- Data Hallucination: Real Representation Regularization constrains same-image foreground-background mixtures toward representations from a non-hallucinating teacher network.The authors report that TriR improves final results.
- Data Hallucination: Inter-class hallucination mixes support foregrounds with all available backgrounds, while Soft Similarity Prior assigns likelihoods using background similarity.Priors are reported to improve inter-class hallucination, especially for one-shot learning; SSP is consistent across protocols.
4. Experiments
SalNet is evaluated on miniImagenet and Open MIC under few-shot protocols, where it generally outperforms state-of-the-art baselines. Ablations and sensitivity studies attribute gains to saliency segmentation, hallucination, regularization, similarity priors, and mask dilation, while detector quality limits performance.
- Approximately 4.4% and 3.3% higher top-1 accuracy than SoSN are achieved by inter-class hallucination on 1- and 5-shot protocols, respectively.
- SalNet improvements on Open MIC are consistent with miniImagenet, but some difficult splits improve by only approximately 1.1% because salient objects are hard to segment.
- Ablation study: Disabling saliency segmentation and hallucination reduces performance on all protocols to SoSN's baseline level.
- Ablation study: SalNet still outperforms SoSN without hallucinated datapoints, suggesting that foreground-background segmentation transfers spatial knowledge for query-support similarity.
- Sensitivity and component studies: TriR provides approximately 1% gain at β = 0.01 over β = 0, with gains up to 1.6% and 1.5% for intra- and inter-class hallucination.
- Sensitivity and component studies: SSP yields a maximum observed accuracy gain of approximately 3.3% and outperforms HSP by up to approximately 2.2%.
- Saliency methods and mask dilation: Complex saliency methods perform equally well, whereas RBD causes significant performance loss because of numerous saliency-detection failures.
5. Conclusions
The paper presents two lightweight hallucination strategies that use saliency-derived foreground-background pairs and end-to-end SalNet training instead of costly GAN-based generation. TriR and similarity-based priors address noisy maps and implausible combinations, while future work considers self-supervised attention and relaxed segmentation.
- The paper presents two lightweight data hallucination strategies for few-shot learning using a readily available saliency network instead of costly GAN-based methods.
- SalNet trains end-to-end on saliency-derived foreground-background pairs to hallucinate additional feature-space training samples.
- TriR regularizes hallucination with viable representations from real foreground-background pairs, while similarity priors reduce the influence of incorrect hypotheses.
- Future work will investigate a self-supervised attention module for similarity perception and relaxations of saliency segmentation methods.
A. Saliency Maps on the Open MIC dataset
Saliency maps reliably filter many Open MIC exhibits, but they are most likely to fail on composite scenes containing numerous exhibits.
- Many Open MIC exhibits can be filtered out reliably using saliency maps.
- Composite scenes containing numerous exhibits are the cases in which saliency maps most likely fail.
- Despite unreliable maps on composite scenes, the mixing network can reduce saliency-map noise.
B. Evaluations for 224×224 pixel images
The network supports larger input images without modification because second-order representations aggregate variable numbers of feature vectors into fixed-size matrices.
- The experiments use 84 × 84 images for fair comparison with other state-of-the-art methods.
- Larger images can be used without modifying the network because second-order representations aggregate variable feature counts into fixed-size matrices.The paper applies 224 × 224 images to demonstrate benefits from larger image size.
C. Network Architecture of Our Baseline Models and Additional Experiments for TriR
The additional experiments compare baseline pipelines with and without saliency segmentation or hallucination and evaluate alternative teacher networks for TriR regularization.
- Network Architecture of Our Baseline Models: Baseline 1 is the original pipeline without saliency segmentation or data hallucination.
- Additional Experiments for TriR: Table 5 reports miniImagenet accuracy for different image sizes and includes sanity-check results with saliency segmentation and hallucination disabled.
- Network Architecture of Our Baseline Models: Baseline 2 uses saliency segmentation but does not hallucinate data by mixing foregrounds with different backgrounds.
- Additional Experiments for TriR: The reported results use baseline 2 as the teacher in TriR regularization, while baseline 1 is also evaluated as an alternative teacher.
- Additional Experiments for TriR: Both TriR teacher networks perform similarly to each other on the miniImagenet evaluation.