Hallucination Improves Few-Shot Object Detection

1. Introduction

• Few-shot detection must identify both where an object is and what it is, and the particularly challenging regime has fewer than three examples for novel classes.The paper frames this setting as common yet challenging in real-world use.Source: §1 · p. 1
• With one example, the variation among positive RPN boxes is too weak to model a novel category’s appearance, while proposal-generation changes alone remain limited.CoRPNs can provide informative proposals, but their positive novel-class boxes remain similar to the few available instances.Source: §1 · p. 2
• The hallucinator learns shared within-class variation on data-rich base classes and generates novel-class examples in RoI feature space for classifier training.The approach transfers variation such as camera pose, lighting, or articulation, which can generalize across categories.Source: §1 · p. 2
• Training proceeds in an EM-like sequence: fit a classifier on base data, train the hallucinator under its guidance, then refine the classifier with hallucinated novel-class examples.This avoids the complicated meta-learning procedure used by an earlier few-shot classification hallucinator.Source: §1 · p. 2
• The proposed data-hallucination approach significantly outperforms the state-of-the-art TFA [38] and most recent cooperating RPN’s [48] detectors in the extremely-few-shot detection regime.Source: §1 · p. 2

2. Related Work

• Serial detectors generate region proposals before classifying them, whereas parallel detectors perform proposal and classification processes simultaneously, usually trading accuracy for speed.The paper focuses mainly on serial detectors because they achieve state-of-the-art few-shot detection performance, while proposing hallucination as a general strategy that may also apply to parallel detectors.Source: §2 · p. 2
• Few-shot detection research improves feature representations, modifies fine-tuning, or uses meta-learning to adapt features and classifiers from limited examples.Approaches include metric learning, regularized or two-stage fine-tuning, and meta-learned feature reweighting or classifiers.Source: §2 · pp. 2–3
• Proposal-focused methods improve class-aware features or proposal ranking, and RPN ensembles seek to avoid missing informative boxes.Other approaches use contrastive-aware encodings, semantic relations, or multi-scale representations.Source: §2 · p. 3
• Earlier data-hallucination work largely addresses classification, while this paper adapts feature-space hallucination to few-shot detection and its hallucination procedure.The authors state that, to their knowledge, this is the first demonstration of data hallucination’s effectiveness for few-shot detection.Source: §2 · p. 3

3.1. Few-Shot Object Detection Models

• The approach targets TFA and CoRPNs, which share two-stage base/novel fine-tuning but use different proposal-generation mechanisms.TFA fine-tunes its RoI classifier and box regressor on novel-class instances; CoRPNs use multiple cooperating RPNs.Source: §3 · p. 3, §3.1 · p. 3

3.2. Few-Shot Object Detection with Hallucination

• A hallucinator transfers shared within-class variation from base classes into additional novel-class RoI examples for classifier training.It takes training examples as input, and hallucinated examples are appended to original examples; the box regressor is not affected.Source: §3.2 · p. 4, Figure 4 · p. 4
• The hallucinator uses a class prototype and per-example noise, and the classifier's cross-entropy loss on hallucinated examples trains it to produce examples that agree with the classifier.The prototype is the mean of instances in the class and implicitly regularizes the hallucinator against copying seed examples.Source: §3.2 · p. 4

3.3. Training the Hallucinator and the Classifier

• EM-style training alternates between updating the hallucinator with the classifier frozen and updating the classifier and box regressor with the hallucinator frozen.The classifier update uses original and hallucinated examples, while the regressor uses only original examples.Source: §3.3 · p. 4, Figure 5 · p. 4
• The hallucinator is first trained on base classes in a fixed pre-trained RoI feature space, then fine-tuned on base and novel examples while replacing background examples with generated ones.The classifier–hallucinator procedure runs for one or two iterations; additional iterations did not further improve performance.Source: §3.3 · p. 5

4.1. Implementation Details

• Experiments use standard few-shot benchmarks and splits, Faster R-CNN with a ResNet-101 backbone, and a two-layer MLP hallucinator.The hallucinator takes prototype, seed-example, and noise inputs; novel-class prototypes are updated using training and generated examples.Source: §4.1 · pp. 5–6

4.2. Main Results

• On PASCAL VOC, hallucination significantly improves TFA and CoRPNs in 1- and 2-shot settings (AP50 across three splits), while maintaining comparable results to baselines at higher shots.Source: Table 1 · p. 6
• Hallucination slightly degrades base-class performance after fine-tuning, but yields a better base/novel trade-off than MPSR and FsDetView.The authors suggest a smaller fine-tuning learning rate for base-class classifier components as a possible way to address the degradation.Source: §4.2 · p. 7, Table 3 · p. 7

4.3. Ablation Studies

• EM-style training significantly outperforms joint training, and a second EM iteration improves performance while further iterations do not.The comparison uses the same hallucinator architecture; the second-iteration gain is reported across all PASCAL VOC base/novel splits.Source: §4.3 · p. 7, Table 4 · p. 7
• An aggressive hallucinator improves TFA on PASCAL VOC novel sets 2 and 3 without hurting set 1, while the conservative variant performs comparably on sets 2 and 3.The aggressive version hallucinates before the box head and is jointly trained with model components, with an additional cosine prototypical-network loss.Source: §4.3 · pp. 7–8, Table 5 · p. 8
• Increasing hallucinated examples from 0 to 20 improves performance before it saturates and drops slightly; the drop is within the reported 95% confidence interval.The authors note that even a small per-batch count, such as 3, accumulates over many training iterations.Source: §4.3 · p. 8, Table 6 · p. 8
• With fully connected classifiers, hallucination improves CoRPNs in all three PASCAL VOC 1-shot splits, indicating effectiveness beyond cosine classifiers.The compared CoRPNs models use the same hyperparameters and RPN outputs.Source: §4.3 · p. 8, Table 7 · p. 8
• On PASCAL VOC, hallucination increases true positives in most cases and decreases false positives in all cases, with larger improvements in false positives.The authors offer close-to-boundary generated examples and improved decision boundaries as a possible explanation.Source: §4.3 · p. 8, Table 8 · p. 8

• Data hallucination can be incorporated into different detector types and substantially improves few-shot detection.Source: §5 · p. 8
• The method generates additional feature-level training examples to address the lack of sample variation in few-shot detection.Source: §5 · p. 8
• Different design choices change the behavior of the hallucinator.Source: §5 · p. 8
• The method is one instantiation; architecture refinements and more dedicated training control may yield a more effective hallucinator, while extension beyond detection remains a promising direction.Source: §5 · p. 8