PaperlayerView exact revision on arXiv

Source-linked AI summary

Retrieval Augmented Classification for Long-Tail Visual Recognition

Alexander Long, Wei Yin, Thalaiyasingam Ajanthan, Vu Nguyen, Pulak Purkait, Ravi Garg, Alan Blair, Chunhua Shen, Anton van den Hengel

arXiv:2202.11233v1cs.CV

TL;DR

Long-tail visual recognition is difficult because rare classes are underrepresented, while large pretrained models store knowledge implicitly and are costly to fine-tune. RAC adds a parallel retrieval branch over explicit external memory to standard image classification, achieving state-of-the-art improvements while using the training datasets themselves as the information source.

  • Problem

    Long-tail visual recognition remains difficult because sparse tail classes are underrepresented and large models store knowledge implicitly, limiting post-hoc modification and interpretability.

  • Method

    RAC jointly trains a standard base encoder and a retrieval branch that queries pre-encoded images with associated text from explicit non-parametric memory.

  • Results

    RAC improves state-of-the-art long-tail classification results on iNaturalist and Places365-LT by a large margin, with the retrieval branch learning high tail-class accuracy without prompting.

  • Takeaways & Limitations

    Explicit external memory lets pretrained models support inference without fine-tuning while the retrieval and base branches specialize toward tail and common classes, respectively.

  • Takeaways & Limitations

    The evaluation does not cover balanced datasets or additional long-tailed datasets, and label-only retrieval is limited by the CLIP text encoder’s 76-token cap.

Abstract

from arXiv · show

We introduce Retrieval Augmented Classification (RAC), a generic approach to augmenting standard image classification pipelines with an explicit retrieval module. RAC consists of a standard base image encoder fused with a parallel retrieval branch that queries a non-parametric external memory of pre-encoded images and associated text snippets. We apply RAC to the problem of long-tail classification and demonstrate a significant improvement over previous state-of-the-art on Places365-LT and iNaturalist-2018 (14.5% and 6.7% respectively), despite using only the training datasets themselves as the external information source. We demonstrate that RAC's retrieval module, without prompting, learns a high level of accuracy on tail classes. This, in turn, frees the base encoder to focus on common classes, and improve its performance thereon. RAC represents an alternative approach to utilizing large, pretrained models without requiring fine-tuning, as well as a first step towards more effectively making use of external memory within common computer vision architectures.

1. Introduction

RAC addresses long-tail visual recognition by adding explicit external memory to standard classification, separating retrieval of sparse-class knowledge from parametric modeling. It improves benchmark performance while using dataset images and associated text as retrieval information.

  • 1. Introduction: Long-tail recognition is difficult because few-shot tail classes are underrepresented while abundant head classes dominate learning.The tail contains relatively few samples, whereas head classes contain most samples, producing performance gaps relative to balanced data.
  • 1. Introduction: RAC introduces retrieval-augmented classification, explicitly storing tail knowledge in a retrieval-based addition to standard classification pipelines.The approach is motivated by the difficulty of representing long-tailed visual data entirely within model parameters.
  • 1. Introduction: RAC combines a standard base encoder with a multimodal retrieval branch that queries encoded images and associated text from external memory.The retrieval branch uses image representations as keys and can return labels, descriptions, or captions; the simplest index uses the dataset itself.
  • 1. Introduction: The two branches are jointly trained so retrieval handles sparse classes while the base encoder can focus more effectively on common classes.The paper reports that the retrieval branch learns this specialization without explicit prompting.
  • 1. Introduction: RAC achieves state-of-the-art performance on common long-tail classification benchmarks and outperforms prior approaches including LACE.The method also enables inference with large pretrained models without fine-tuning, using them for index and retrieval encoding.
  • 1. Introduction: The contribution includes effective external memory for long-tail recognition, a method improving current state of the art, and reimplemented baselines that also exceed prior results.The paper presents this as an initial demonstration of external memory within long-tail visual recognition.

2. Related Work

Prior work addresses long-tail recognition mainly through resampling, reweighting, logit or loss adjustment, and model ensembling. Related external-memory methods retrieve information for other tasks, whereas RAC retrieves text attached to visually similar samples.

  • 2. Related Work: Long-tail classification methods commonly rebalance training through over-sampling, under-sampling, augmentation, clustering, or distribution-aware weighting.These approaches aim to reduce distribution bias by emphasizing sparse classes or reducing the influence of head classes.
  • 2. Related Work: Other approaches modify logits or losses to align optimization with class-balanced evaluation, while ensemble methods combine models trained under different objectives or sampling schemes.Ensembling can also combine decorrelated classification heads with a routing mechanism for efficient inference.
  • 2. Related Work: External-memory research includes neural memory architectures and latent retrieval, but RAC returns text attached to retrieved samples rather than only the nearest retrieved representation.This distinguishes RAC from textual open-domain retrieval and relates it to fact-memory retrieval for knowledge-intensive question answering.

3. Method

RAC combines a standard classification branch with a retrieval branch that queries precomputed image embeddings and associated text, then combines their normalized logits. The method uses long-tail objectives and an approximate k-NN memory to retrieve textual evidence for each query.

  • Preliminaries and LACE Loss: RAC uses a long-tail formulation in which class frequencies define the training imbalance and balanced error is optimized through reweighting or logit adjustment.The paper describes BalCE as a strong baseline and identifies LACE and LDAM as special cases of the general adjusted or reweighted softmax loss.
  • Retrieval Augmented Classification: RAC splits classification into base and retrieval branches, combining their normalized logits under a common LACE loss.The combined logits are rescaled to preserve training dynamics, while branch outputs remain individually interpretable.
  • Retrieval Module: The base branch uses a standard backbone, while the retrieval branch queries a precomputed embedding index and encodes text attached to the nearest images.The retrieval module uses a frozen pretrained image encoder for keys and a text encoder to produce fixed-length retrieval logits.
  • Retrieval Module: Each query retrieves k nearest images by cosine similarity, recovers their associated text, and encodes the resulting batch into retrieval logits.Text is truncated to 76 tokens and zero-padded so one text-encoder call processes the batch.
  • Retrieval Module: The retrieval memory can incorporate labels, descriptions, or captions, and datasets can be added or removed without fine-tuning the classifier.The authors present this as an alternative way to incorporate new information when similar examples enter the index.

4. Experiments

RAC is evaluated on long-tail benchmarks using pretrained encoders and dataset-derived indexes, with ablations examining retrieval behavior, text encoding, retrieved-neighbor count, index content, and computational cost.

  • 4.1. Places365-LT: The retrieval network skews accuracy toward few-shot classes without explicit prompting, while the base encoder is trained jointly under the common LACE loss.This learned division resembles the hard-coded ensembling strategy used in TADE.
  • 4.2. iNaturalist-2018: 21.6%: using 16 × 16 patches boosts retrieval accuracy on iNat, likely because its species labels and images are fine-grained.Results are also reported at 384 × 384 resolution in addition to the commonly studied 224 × 224 setting.
  • 4.3. Ablation: 8.04% and 7.72%: RAC increases all-class top-1 accuracy over BalCE on Places365-LT and iNat, respectively.It also improves over standard cross entropy by 25.48% on Places365-LT.
  • 4.3. Ablation: 30.42% and 10.98%: RAC improves tail-class accuracy over BalCE on Places365-LT and iNat, respectively.
  • 4.6. Effect of k: Increasing k improves retrieval-only accuracy until the 76-token text truncation cutoff, while training time rises linearly; experiments use k = 30.The retrieval text encoder learns to disregard common classes even when additional retrieved samples cannot be the correct few-shot class.
  • 4.3. Ablation: ViTs significantly outperform ResNets for standard retrieval despite being trained on the same data, making them critical to RAC’s performance.The retrieval experiment encodes training images and predicts each test image using the closest indexed image under exact-match evaluation.

5. Limitations

RAC performs robustly on the two evaluated long-tailed distributions, but its scope and retrieval content impose clear boundaries that motivate further work.

  • RAC’s analysis should expand to additional long-tailed datasets and balanced datasets, which were not explored.The paper identifies both broader long-tail evaluation and balanced-data performance as open directions.
  • The retrieved labels provide limited information and cap RAC’s performance, while richer captions or paragraphs are constrained by the CLIP text encoder’s 76-token limit.The authors leave expanding the text capacity and retrieving richer content for future work.

6. Conclusion

The paper concludes that RAC is a generic retrieval-augmented classification approach whose retrieval branch handles tail classes while the base encoder focuses on common classes. It reports large improvements on iNat and Places365-LT and identifies broader external-memory applications.

  • RAC’s retrieval module achieves high tail-class accuracy without prompting, freeing the base encoder to focus on common classes.The conclusion presents this division of labor as the central mechanism of RAC.
  • RAC improves state-of-the-art results by a large margin on the iNat and Places365-LT long-tail image-classification benchmarks.
  • The paper presents RAC as a step toward using external memory in common computer-vision architectures, with possible applications to one/few-shot and continual learning.These applications are stated as predicted future uses rather than evaluated results.

A.1. Re-weighting and Temperature Scaling

The appendix examines re-weighting and temperature scaling for RAC’s top-level loss. Re-weighting improves Places365-LT overall and especially tail-class performance, whereas temperature scaling provides only a minor additional effect.

  • Re-weighting and Temperature Scaling: RAC uses unscaled logit adjustment with no re-weighting and τ = 1 as its top-level loss, although other work reports empirical gains from combining adjustment with re-weighting.
  • Re-weighting and Temperature Scaling: Re-weighting increases overall RAC accuracy on Places365-LT, with the largest improvement appearing for tail classes, but this trend does not hold on iNat.The comparison concerns both re-weighting schemes evaluated with logit adjustment.
  • Re-weighting and Temperature Scaling: Slightly higher τ increases overall Places365-LT accuracy, but the effect is minor compared with re-weighting.The sweep is summarized by Figure S1.
  • Re-weighting and Temperature Scaling: The limited benefit of temperature scaling on Places365-LT may reflect the non-separability of many classes caused by high label noise.

A.2. Index Ablations

Index ablations show that distance-metric choice has little effect, while approximate HNSW search slightly reduces accuracy but can substantially accelerate lookup on large indexes.

  • Index Ablations: Query time includes ViT-B-16 sample encoding, which is the primary overhead on small indexes.
  • Index Ablations: Approximate HNSW search causes only a minor accuracy drop while providing a 2× speedup on large indexes.The speed advantage becomes important as index size grows, when lookup can otherwise bottleneck training.
  • Index Ablations: The choice between ℓ2 and cosine distance has little effect on accuracy, although cosine adds minor normalization overhead.

A.3. Per-class Accuracy on iNat

On iNaturalist2018, the retrieval branch performs best on tail classes, while the base network focuses largely on many and mid-frequency classes. The per-class trend remains visible despite sparse validation samples per class.

  • A.3. Per-class Accuracy on iNat: Three validation samples per iNat class produce a square-wave appearance in the per-class plots, so a moving average clarifies the underlying trend.The figure shows a 300-sample moving average over classes for clarity.
  • A.3. Per-class Accuracy on iNat: The retrieval module performs best on tail classes, while the base network largely focuses on many and mid-frequency classes.This trend remains visible in the sliding-window moving average despite only three validation samples per iNat class.
  • A.3. Per-class Accuracy on iNat: For Places365-LT, final numbers use the test set after development on training and validation splits, whereas iNaturalist2018 results use the released validation split because test labels are unavailable.No validation samples are used during final Places365-LT training.

C. Retrieval Branch Visualization

The retrieval-branch visualization shows query images alongside retrieved labels, distances, frequencies, distance distributions, and correct labels. It uses distance- and frequency-sorted retrieval outputs to inspect how retrieved snippets support classification.

  • C. Retrieval Branch Visualization: Non-exact retrieved snippets can still inform classification because related words such as plane, runway, concrete, sky, and propeller support the airport prediction.The retrieval branch uses k = 30 for all runs, with returned strings visualized for Places365-LT and iNaturalist2018.
  • C. Retrieval Branch Visualization: Each visualization column shows the query image, distance-sorted retrieved labels, frequency-sorted labels, a distance histogram, and the correct label.Exact matches are colored green, and retrieved-label lists are restricted to the top eight.
  • C. Retrieval Branch Visualization: The visualizations cover randomly drawn samples from Places365-LT and iNaturalist2018, with cosine similarity represented so higher values indicate more similar samples.The accompanying figures are S3 for Places365-LT and S4 for iNaturalist2018.
Loading 2202.11233v1…