Source-linked AI summary
Large-Scale Long-Tailed Recognition in an Open World
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, Stella X. Yu
TL;DR
The paper addresses recognition under naturally long-tailed, open-ended data, where systems must handle head, tail, and novel classes together. It introduces dynamic meta-embedding to transfer visual knowledge and calibrate novelty, and reports consistent state-of-the-art performance across three curated benchmarks, with a noted fine-grained discrimination trade-off.
Problem
Open Long-Tailed Recognition requires one integrated algorithm to handle imbalanced classification, few-shot learning, and open-set recognition across head, tail, and open classes.
Method
Dynamic meta-embedding combines a direct image feature with a memory feature, while memory-based calibration indicates familiarity and supports recognition across the class spectrum.
Results
The method consistently outperforms the state-of-the-art on three curated large-scale OLTR benchmarks spanning ImageNet-LT, Places-LT, and MS1M-LT.
Takeaways & Limitations
The proposed benchmarks, code, datasets, and models provide a basis for future OLTR research and applications involving imbalanced, few-shot, and open-set recognition.
Takeaways & Limitations
Feature infusion slightly sacrifices fine-grained discrimination, with confusion reported between many-shot and medium-shot classes.
Abstract
from arXiv · showhide
Real world data often have a long-tailed and open-ended distribution. A practical recognition system must classify among majority and minority classes, generalize from a few known instances, and acknowledge novelty upon a never seen instance. We define Open Long-Tailed Recognition (OLTR) as learning from such naturally distributed data and optimizing the classification accuracy over a balanced test set which include head, tail, and open classes. OLTR must handle imbalanced classification, few-shot learning, and open-set recognition in one integrated algorithm, whereas existing classification approaches focus only on one aspect and deliver poorly over the entire class spectrum. The key challenges are how to share visual knowledge between head and tail classes and how to reduce confusion between tail and open classes. We develop an integrated OLTR algorithm that maps an image to a feature space such that visual concepts can easily relate to each other based on a learned metric that respects the closed-world classification while acknowledging the novelty of the open world. Our so-called dynamic meta-embedding combines a direct image feature and an associated memory feature, with the feature norm indicating the familiarity to known classes. On three large-scale OLTR datasets we curate from object-centric ImageNet, scene-centric Places, and face-centric MS1M data, our method consistently outperforms the state-of-the-art. Our code, datasets, and models enable future OLTR research and are publicly available at https://liuziwei7.github.io/projects/LongTail.html.
1. Introduction
The paper defines Open Long-Tailed Recognition as balanced evaluation across head, tail, and open classes, requiring one system to combine imbalanced, few-shot, and open-set recognition. It proposes dynamic meta-embedding and related mechanisms to transfer knowledge across classes while detecting novelty.
- OLTR evaluates recognition on a balanced test set spanning head, tail, and open classes learned from naturally long-tailed, open-ended data.
- Existing approaches typically target only one aspect of OLTR, leaving integrated recognition across the full class spectrum poorly addressed.
- OLTR must share visual knowledge between head and tail classes while reducing confusion between tail and open classes.
- Dynamic meta-embedding enriches a direct image feature with a memory feature that transfers visual knowledge, particularly to tail classes.
- The method calibrates embedding scale using distance to visual memory, making distant features closer to the origin and more indicative of open-set instances.
- The authors curate ImageNet-LT, Places-LT, and MS1M-LT benchmarks and report consistent state-of-the-art improvements with publicly available code, data, and models.
2. Related Works
The related-work discussion positions OLTR at the intersection of imbalanced classification, few-shot learning, and open-set recognition. Its dynamic meta-embedding combines metric and meta-learning ideas without episodic training, while confidence is learned through feature scaling rather than output-logit calibration.
- OLTR integrates imbalanced classification, few-shot learning, and open-set recognition, which prior work commonly studies in isolation.
- Dynamic meta-embedding combines metric learning, which separates class centroids, with meta-learning, which transfers knowledge to tail classes.
- Unlike many few-shot methods focused on novel classes, OLTR is designed for naturally long-tailed training data and the full class spectrum.
- The memory feature adaptively transfers knowledge to both head and tail classes through a learned concept selector and requires no episodic training.
- Instead of fitting output logits as OpenMax does, OLTR incorporates confidence estimation into feature learning by rescaling meta-embeddings relative to visual memory.
3. Our OLTR Model
The OLTR model combines dynamic meta-embedding for knowledge transfer and open-set sensitivity with modulated attention for head–tail discrimination. It trains these components with metric-aware classification mechanisms and a cosine classifier.
- Dynamic Meta-Embedding: Dynamic meta-embedding combines a direct CNN feature with a memory feature to transfer visual knowledge from head classes to data-poor tail classes.The direct feature is learned from classification loss, while the memory feature retrieves related visual concepts from discriminative class centroids.
- Dynamic Meta-Embedding: The model dynamically scales the combined embedding by inverse distance to the nearest memory centroid, making distant inputs more likely to represent open classes.Small reachability distance produces a larger weight, whereas large distance scales the embedding toward an almost zero vector.
- Dynamic Meta-Embedding: A concept selector adaptively balances direct and memory features, favoring direct features for head classes and memory features for tail classes.The selector uses a lightweight network with tanh activation to make this choice softly.
- Modulated Attention: Modulated attention applies conditional spatial attention to self-attention maps, allowing different classes to select different spatial contexts and preserve head–tail discrimination.The design is applied to the last CNN feature map and was reported to outperform applying spatial attention directly to the input feature map.
- Learning: The final classifier uses normalized meta-embeddings and normalized class weights in a cosine classifier, with a nonlinear squashing function amplifying reachability effects.Training combines cross-entropy classification loss with a large-margin loss between embeddings and centroids; λ is set to 0.1 in the experiments.
4. Experiments
The experiments evaluate OLTR on three curated long-tailed benchmarks and compare module contributions, broad recognition performance, robustness, and failure cases. Across datasets and class-frequency groups, the approach improves performance while retaining a documented fine-grained discrimination trade-off.
- Datasets and Metrics: The study evaluates ImageNet-LT, Places-LT, and MS1M-LT, using closed-set and open-set metrics to assess long-tailed recognition.ImageNet-LT and Places-LT provide object- and scene-centric benchmarks, while MS1M-LT is evaluated on MegaFace without identity overlap.
- Ablation Study: The dynamic meta-embedding and concept selector produce large improvements across many-, medium-, and few-shot classes, while confidence calibration helps distinguish tail classes from open classes.The ablation is reported with open-set top-1 classification accuracy on ImageNet-LT.
- Ablation Study: Modulated attention contributes more to discrimination between many-shot and few-shot classes than medium-shot classes and outperforms directly applied spatial attention.It is designed to support adaptive context selection across feature layers.
- Result Comparisons: On ImageNet-LT, the approach comprehensively improves many-, medium-, few-shot, and open-class performance, whereas competing methods trade head-class accuracy, open-set F-measure, or imbalanced-base-class handling.Figure 6 reports across-the-board absolute F1 gains over the plain model.
- Result Comparisons: On Places-LT, the method consistently outperforms alternatives under closed-set and open-set settings, with an even more pronounced advantage under F-measure.The comparison uses a pre-trained ResNet-152 baseline.
- Result Comparisons: 58.7 is the reported result for Ours, exceeding MetaModelNet at 57.3, Model Reg. at 54.7, Cost-Sensitive at 52.4, and Plain Model at 48.0.These values are listed for the benchmark comparison in the supplied results passage.
- Result Comparisons: On MS1M-LT, the largest gains occur for one-shot identities at 3.0% and zero-shot identities at 1.8%.Testing uses MegaFace, with pseudo-shot subsets estimated from similarity to training samples.
- Result Comparisons: The method achieves a 1.4% improvement over the prior best on SUN-LT while requiring less computational cost than MetaModelNet’s recursive training procedure.The approach transfers visual knowledge among features rather than learning a series of classifier transformations.
5. Conclusions
The paper introduces OLTR and an integrated dynamic meta-embedding algorithm, validating it on three curated large-scale benchmarks and releasing code and data for future research.
- OLTR learns from natural long-tail, open-ended data and evaluates balanced accuracy across head, tail, and open classes.
- Dynamic meta-embedding shares visual knowledge between head and tail classes while reducing confusion between tail and open classes.
- The method is validated on ImageNet-LT, Places-LT, and MS1M-LT, with publicly available code and data intended to enable future research.
Appendices
The appendices provide intuitive explanations, component effects, and additional methodology and visualization details for the proposed open long-tail recognition approach.
- The supplementary material includes an intuitive explanation, methodology details, experimental setup, and additional visualization of the approach.
- Head, tail, and open classes form a continuous spectrum from the knowledge gained through training observations.
- The approach aggregates head- and tail-class knowledge into visual memory, infuses associated memory features into direct features, and calibrates open-class confidence by memory reachability.
- Table 5 summarizes the effects of the individual components in the approach.
B. Relation to Fairness Analysis
Open long-tail recognition and fairness analysis both address imbalance and transferable representations, but differ in the categories and objectives they target.
- Both open long-tail recognition and fairness analysis address imbalance in real-world data, but OLTR covers long-tailed known and unknown categories.
- OLTR optimizes overall accuracy across categories, whereas fairness analysis optimizes several attribute-wise criteria.
- Preliminary Table 4 results suggest that dynamic meta-embedding may also be promising for fairness analysis.
C. More Methodology Details
The supplementary methodology details describe notation, discriminative-centroid construction, and the loss functions used to train dynamic meta-embeddings.
- The paper summarizes its notation in Table 7.
- Figure 10 presents the dataset statistics of ImageNet-LT.
- Discriminative centroids are obtained through a step-by-step procedure illustrated in Figure 11.
- The cross-entropy loss compares dynamic meta-embeddings with ground-truth category labels using a cosine classifier.
- The large-margin loss pulls each embedding toward its group centroid and pushes it away from centroids of other groups.
D.1. Open Long-Tail Dataset Preparation
The paper prepares long-tailed datasets spanning object, scene, and face recognition, with evaluation and training procedures tailored to class imbalance and shot variation.
- Dataset construction: ImageNet-LT uses a Pareto-distributed training set with 1,280∼5 images per class across 1,000 ImageNet classes.The classes are randomly split into 389 base and 611 novel classes.
- Dataset construction: MegaFace evaluates face identification by matching 3,530 FaceScrub probe images against a 1M-image gallery.Identification rate is the mean hit rate, with test subsets partitioned by estimated many-shot, few-shot, one-shot, and zero-shot occurrences.
- Dataset construction: SUN-LT contains 1,132∼1 training images and 40 testing images per class, plus five validation images sampled from unused training data.
- Preprocessing: Training images are resized to 256 × 256, randomly cropped to 224 × 224, horizontally flipped with probability p = 0.5, and color-jittered.Validation and testing use center crops to 224 × 224 without further augmentation.
- Training protocols: ImageNet-LT uses a randomly initialized ResNet-10, while Places-LT and SUN-LT use two-stage training with an ImageNet-pretrained ResNet-152.MS1M-LT uses an ImageNet-pretrained ResNet-50 with a linear classifier and cross-entropy loss.
D.4. Evaluation Protocols
The evaluation protocols measure closed-set classification under open-class contamination and use F-measure components to quantify closed-set precision and open-set sensitivity.
- Classification metrics: Top-1 classification accuracy is the mean accuracy over balanced close-set categories while open classes are treated as one unknown class.Predictions use the class with the highest softmax probability.
- F-measure: F-measure combines precision and recall as two times their product divided by their sum.
- F-measure: Precision uses true positives among closed-testing predictions and false positives from incorrect closed-testing predictions.
- F-measure: Recall uses true positives divided by true positives plus false negatives, where false negatives are open-set images predicted as known categories.
- Qualitative analysis: Memory-feature visualizations report discriminative visual traits for Places-LT scenes and infused visual concepts for MS1M-LT faces.