Source-linked AI summary
Few-Shot Classification with Feature Map Reconstruction Networks
Davis Wertheimer, Luming Tang, Bharath Hariharan
TL;DR
Few-shot classification is challenging when labeled images are scarce or expensive to obtain. FRN reconstructs query feature maps from class support features using closed-form ridge regression, achieving state-of-the-art fine-grained performance and competitive general-benchmark results. However, decoder reconstructions can fall off the input manifold under heavy regularization.
Problem
Few-shot classifiers must learn novel categories despite limited labeled images, which may be rare, expensive to annotate, or unavailable during rapid deployment.
Method
FRN scores class membership by reconstructing query feature maps from pooled support features, solving the reconstruction weights through closed-form ridge regression.
Results
FRN achieves state-of-the-art performance on four fine-grained few-shot benchmarks and highly competitive performance on general few-shot recognition benchmarks.
Takeaways & Limitations
Closed-form feature-map reconstruction yields a straightforward classifier that incorporates fine spatial details without overfitting to position or pose.
Takeaways & Limitations
Heavy regularization can push reconstructions off the decoder's input manifold, producing uniformly flat grey-brown images.
Abstract
from arXiv · showhide
In this paper we reformulate few-shot classification as a reconstruction problem in latent space. The ability of the network to reconstruct a query feature map from support features of a given class predicts membership of the query in that class. We introduce a novel mechanism for few-shot classification by regressing directly from support features to query features in closed form, without introducing any new modules or large-scale learnable parameters. The resulting Feature Map Reconstruction Networks are both more performant and computationally efficient than previous approaches. We demonstrate consistent and substantial accuracy gains on four fine-grained benchmarks with varying neural architectures. Our model is also competitive on the non-fine-grained mini-ImageNet and tiered-ImageNet benchmarks with minimal bells and whistles.
1. Introduction
Few-shot classification must adapt to unseen classes from very few labeled images while preserving spatial detail without encoding nuisance pose. FRN addresses this by reconstructing query feature maps from class-specific support features using closed-form ridge regression, achieving strong results across fine-grained and general benchmarks.
- Few-shot classifiers must rapidly fit novel, possibly unseen classes from a small number of reference images.
- Metric-learning methods replace a parametric classifier head with a class-agnostic latent-space distance function.
- Feature-map methods face a representation tradeoff: average pooling discards spatial detail, while flattening preserves location sensitivity and pose information.
- FRN pools support feature vectors across images and locations, reconstructs each query feature-map location with weighted support features, and scores classes by negative average squared reconstruction error.
- FRN formulates reconstruction as ridge regression, yielding a closed-form solution with only a single learned soft constraint instead of iterative procedures or large attention modules.
- FRN demonstrates across-the-board superiority on four fine-grained datasets and competitive performance on mini-ImageNet and tiered-ImageNet with Conv-4 and ResNet-12 architectures.
3. Method
FRN predicts class membership by reconstructing query feature maps from class-specific support features using closed-form ridge regression. The method learns only three parameters while supporting efficient batch computation and a specialized pre-training scheme.
- Feature Map Reconstruction: FRN pools spatial features from each class’s support images and reconstructs every query feature-map location as a weighted sum of those support features.The support features form Sc ∈ R^(kr×d), while the reconstruction uses WSc ≈ Q.
- Feature Map Ridge Regression: The optimal reconstruction weights solve a ridge-regression problem, with λ regularizing over- or under-constrained systems.The ridge penalty makes the linear system tractable when kr ≠ d.
- Classification Objective: Class logits use the negative mean squared Euclidean reconstruction error across feature-map locations, with a learnable temperature factor γ.Predicted query probabilities are trained with cross-entropy during episodic meta-training.
- Learning the Degree of Regularization: FRN learns λ through meta-learning and introduces ρ to separate regularization strength from reconstruction magnitude.λ and ρ are parameterized as e^α and e^β to ensure non-negativity; increasing ρ alongside λ can penalize large weights without collapsing reconstructions to the origin.
- Efficiency: FRN introduces only three learned parameters: α, β, and γ, and computes reconstructions for query batches using one support-matrix calculation per episode class.The formulation can use either a kr × kr or d × d matrix inversion depending on the relationship between kr and d; the alternatives are algebraically equivalent.
- Pre-Training: FRN’s pre-training scheme preserves feature reconstruction as the classification logit by using class-specific dummy feature maps, then discards those learned matrices after pre-training.Standard average-pooled classifier pre-training does not produce the spatially distinct feature maps FRN requires.
4. Experiments
Experiments evaluate FRN across fine-grained, general, and cross-domain few-shot benchmarks using Conv-4 and ResNet-12 backbones. FRN shows consistent gains on fine-grained tasks, competitive general recognition, and strong cross-domain performance.
- Experimental setup: FRN is evaluated with Conv-4 and ResNet-12 backbones in standard 5-way, 1-shot and 5-shot settings.Accuracy and 95% confidence intervals are measured over 10,000 trials.
- Fine-Grained Few-Shot Classification: Fine-grained experiments cover cropped and uncropped CUB, Aircraft, meta-iNat, and tiered meta-iNat under fair baseline comparisons without pre-training.The implemented baselines are ProtoNet, CTX, and DSN, with FRN hyperparameters not separately tuned.
- Fine-Grained Few-Shot Classification: 2-7 points: FRN improves accuracy over the nearest baseline in all 1-shot fine-grained settings, with a mean gain of 3.5 points.Results correspond to cropped CUB, uncropped CUB, Aircraft, and combined meta-iNat and tiered meta-iNat evaluations.
- Fine-Grained Few-Shot Classification: Re-implemented baselines are competitive with, and sometimes outperform, previously published results, supporting the fairness of the fine-grained comparisons.The authors conclude that FRN is broadly effective for fine-grained few-shot classification.
- General Few-Shot Classification: On mini-ImageNet and tiered-ImageNet, FRN is highly competitive with recent state-of-the-art results while using pre-training without extra techniques or inference-time gradient finetuning.The general few-shot experiments compare standard benchmark settings and use pre-training followed by episodic finetuning.
- Cross-Domain Few-Shot Classification: In the mini-ImageNet→CUB cross-domain setting, FRN outperforms previous methods by a wide margin across multiple CUB test-class splits.The evaluation includes the split from prior work, the split used for the fine-grained experiments, and all 200 CUB classes.
5. Analysis
FRN’s analysis shows that training-shot choice and pre-training strategy materially affect performance, while same-class reconstructions better preserve image information than different-class reconstructions. Regularization effects vary by architecture, and visual regeneration supports the semantic faithfulness of the learned reconstructions.
- Training shot: 5-shot-trained FRN consistently outperforms 1-shot-trained FRN even on 1-shot evaluation.The reported main-paper results therefore use 5-shot-trained models, while 1-shot training is treated as an ablation.
- Pre-training: Two-round pre-training followed by episodic fine-tuning appears crucial for competitive general few-shot performance.Pre-training alone does not produce a competitive few-shot learner, and a scratch-trained FRN can outperform a naively evaluated pre-trained FRN.
- Pre-training: Average-pooled pre-training does not produce a viable classifier because its features are insufficiently spatially distinct for FRN fine-tuning to recover a meaningful feature space.The resulting classifier is worse than one trained from scratch.
- Regularization: The auxiliary loss has little to no impact, whereas learning λ and ρ benefits Conv-4 but not ResNet-12.The ResNet-12 feature space may already be rich enough to overcome regularization issues.
- Reconstruction analysis: Same-class feature-map reconstructions yield smaller increases in regenerated-image pixel error than different-class reconstructions.Figure 3 and Table 10 support greater semantic faithfulness for same-class reconstructions, although ground-truth-feature regenerations also discard class-irrelevant details.
6. Conclusion
FRN reformulates few-shot classification as latent-space query-feature reconstruction. Its closed-form classifier preserves fine spatial detail without overfitting to position or pose, achieving state-of-the-art results on four fine-grained benchmarks and highly competitive general performance.
- 6. Conclusion: FRN classifies queries by reconstructing their feature maps in latent space using a closed-form solution.The approach is described as straightforward and powerful.
- 6. Conclusion: FRN incorporates fine spatial details without overfitting to position or pose.The conclusion presents this as the central representational advantage of the reconstruction-based classifier.
- 6. Conclusion: FRN achieves state-of-the-art performance on four fine-grained few-shot classification benchmarks and highly competitive performance in the general setting.
7. Pseudo-Codes for Calculating ¯Qc
The pseudo-code computes reconstructed query features using batched matrix operations and a regularized support-derived projection. Two equivalent formulations are selected according to whether the feature dimension or support count is larger.
- Eq. 8 implementation: The alternative formulation projects through a regularized inverse in support space and is more computation-efficient when kr > d.The implementation uses this formula consistently in the experiments when applicable.
- Inputs and output: The implementation accepts query and support feature maps together with λ and ρ, then returns reconstructed query features Q̄.The query tensor has shape [way*query_shot*r, d], while support has shape [way, support_shot*r, d].
- Eq. 10 implementation: The first formulation constructs a regularized support covariance inverse, forms a projection, and scales the projected query by ρ.The calculation is parallelized through batched matrix multiplication and inversion.
8. Ablation Studies
Ablations show that FRN’s auxiliary loss contributes little, while the value of learned regularization depends on network capacity. The study also compares one-shot and five-shot training and reports the evaluation setup used for these analyses.
- Training-shot ablation: 5-shot-trained FRN consistently outperforms FRN trained with 1-shot episodes, even on 1-shot evaluation.The main paper reports the superior 5-shot-trained results and places 1-shot-trained performance in Table 11.
- Auxiliary loss: The auxiliary loss has little to no consistent impact on FRN performance.It is retained in experimental models for apples-to-apples comparisons with baselines that use it.
- Study setup: The ablation study evaluates regularization parameters and auxiliary loss on the cropped CUB benchmark.
- Regularization: Learning λ and ρ helps the 4-layer network, while ResNet-12 benefits slightly from removing them.The ResNet-12 difference may be small or insignificant because its more powerful feature space can overcome regularization issues.
9. Training Details
The experiments use standardized preprocessing, backbones, optimization settings, and episode configurations across several benchmarks, with limited model-specific tuning. FRN’s main benchmark results use models trained with 5-shot episodes.
- Pre-processing: Training uses random cropping, resizing, color jittering, and horizontal flipping, with dataset-specific input resolutions for training and inference.Aircraft and CUB use bounding-box crops in some inference settings, while other datasets use raw images or alternate preprocessing pipelines.
- Network Backbones: Experiments use Conv-4 and ResNet-12 backbones, differing in depth, residual structure, channel widths, and pooling configuration.Conv-4 has four convolutional layers, whereas ResNet-12 uses four residual blocks with three convolutions each.
- Optimization: Unless otherwise stated, models use weight decay of 5e-4, shared hyperparameters across baselines and FRN, and dataset-specific measures for training stability.The authors avoid separately tuning FRN beyond addressing obvious instability cases.
- Normalization: Baseline logits are temperature-scaled and normalized, while FRN uses different embedding normalization because regularization makes logit scale poorly aligned with embedding scale.For FRN ResNet-12, stable training requires downscaling embeddings by a factor of 640.
- Episode Setup: Main-paper FRN results come from models trained with 5-shot episodes; 1-shot FRN training is restricted to ablation studies.The reported episode way and query count vary by backbone to reduce memory usage.
- Benchmark Training: The benchmark protocol includes long SGD or Adam schedules, validation-based checkpoint selection where available, and final-model selection for meta-iNat variants without validation sets.Learning-rate schedules and training durations are specified separately for CUB, Aircraft, meta-iNat, mini-ImageNet, and tiered-ImageNet.
10. Computational Efficiency
FRN is substantially more computationally efficient than DeepEMD and mitigates memory growth relative to CTX as feature-map resolution increases. Its two algebraically equivalent implementations can be selected according to runtime behavior.
- Speed: FRN is much more efficient than DeepEMD in both training and evaluation on mini-ImageNet with ResNet-12 backbones.The comparison varies episode shot numbers and implementation variants.
- Speed: DeepEMD’s training path depends on iterative transport solvers, and the faster OpenCV solver cannot provide gradients suitable for meta-training.The passage contrasts OpenCV’s modified Simplex algorithm with qpth’s interior-point method.
- FRN Implementations: For ResNet-12 with d = 640, k = 1 or 5, and r = 25, Eq. 8 is expected and observed to be more efficient than Eq. 10.The analysis applies because d > kr, although other experiments use Eq. 10 for consistency.
- FRN Implementations: The predicted efficiency ordering can fail in particular GPU implementations, so Eq. 8 and Eq. 10 remain interchangeable algebraically and can be substituted dynamically.In one 5-shot 5×5 setting, Eq. 8 is more efficient despite inverting a larger matrix.
11. Additional Results on CUB
Additional CUB experiments show that FRN performance is robust to class-split choice and improves slightly with pre-training, though the pre-training gain is smaller than on mini-ImageNet.
- Class Splits: FRN performance is not impacted by the choice between the evaluated CUB class splits.The uncropped-image experiment uses the class split from Chen et al. [7] and reports results averaged over 10,000 trials with 95% confidence intervals.
- Pre-Training: Pre-training slightly improves final CUB accuracy compared with training from scratch.The comparison uses FRN with a ResNet-12 backbone and raw images as input.
- Pre-Training: The pre-training gain is smaller on CUB than on mini-ImageNet because CUB is only about 1/6 the size of mini-ImageNet.The authors characterize episodic-from-scratch optimization as comparatively easier on CUB.
12. Additional Visualizations
Additional visualizations compare target, autoencoded, same-class reconstructed, and different-class reconstructed images for CUB and mini-ImageNet. Same-class reconstructions are more faithful than different-class reconstructions, especially in shape.
- Overview: The visualizations provide additional reconstruction trials corresponding to the main paper’s Figure 3.They extend the qualitative comparison across same-class and different-class support images.
- CUB: CUB visualizations place support images on the left and target, autoencoded, same-class, and different-class reconstructions in successive rows on the right.Same-class reconstructions are clearly superior to those from different classes.
- mini-ImageNet: mini-ImageNet visualizations use the same row organization, with same-class reconstructions tending to gray out or darken colors while preserving shape more faithfully.The comparison emphasizes shape fidelity rather than color preservation.