PaperlayerView exact revision on arXiv

Source-linked AI summary

Visual Recognition with Deep Nearest Centroids

Wenguan Wang, Cheng Han, Tianfei Zhou, Dongfang Liu

arXiv:2209.07383v2cs.CV

TL;DR

Existing visual recognition models rely on opaque parametric classifiers that do not explicitly model latent class structure. DNC instead uses distance-based classification over clustered class sub-centroids while learning only the feature representation, achieving stronger image and pixel recognition with improved transferability and interpretability. Its main scope boundary is that the current centroid count is fixed across classes, although varying it may improve performance.

  • Problem

    Parametric classifiers have abstract, non-transparent parameters, do not directly supervise the representation, and constrain output dimensionality to the number of classes.

  • Method

    DNC clusters training data within each class into sub-centroids and classifies queries by proximity to these representatives while optimizing the feature representation.

  • Results

    DNC outperforms parametric counterparts on image classification and improves semantic segmentation across reported datasets, architectures, and models, while enabling exemplar-based interpretability.

  • Takeaways & Limitations

    DNC provides a simpler, more transparent, transferable, and explainable framework for large-scale visual recognition.

  • Takeaways & Limitations

    The number of class centroids K is fixed identically across classes, despite differing intra-class variability.

Abstract

from arXiv · show

We devise deep nearest centroids (DNC), a conceptually elegant yet surprisingly effective network for large-scale visual recognition, by revisiting Nearest Centroids, one of the most classic and simple classifiers. Current deep models learn the classifier in a fully parametric manner, ignoring the latent data structure and lacking simplicity and explainability. DNC instead conducts nonparametric, case-based reasoning; it utilizes sub-centroids of training samples to describe class distributions and clearly explains the classification as the proximity of test data and the class sub-centroids in the feature space. Due to the distance-based nature, the network output dimensionality is flexible, and all the learnable parameters are only for data embedding. That means all the knowledge learnt for ImageNet classification can be completely transferred for pixel recognition learning, under the "pre-training and fine-tuning" paradigm. Apart from its nested simplicity and intuitive decision-making mechanism, DNC can even possess ad-hoc explainability when the sub-centroids are selected as actual training images that humans can view and inspect. Compared with parametric counterparts, DNC performs better on image classification (CIFAR-10, ImageNet) and greatly boots pixel recognition (ADE20K, Cityscapes), with improved transparency and fewer learnable parameters, using various network architectures (ResNet, Swin) and segmentation models (FCN, DeepLabV3, Swin). We feel this work brings fundamental insights into related fields.

1 INTRODUCTION

DNC revisits Nearest Centroids for visual recognition, replacing opaque parametric classification with distance-based reasoning over learned class sub-centroids. It combines within-class pattern discovery with supervised representation learning and reports stronger performance, transferability, and interpretability across recognition tasks.

  • Motivation: Parametric softmax classifiers use abstract class-specific parameters, making their decisions difficult to explain and potentially overlooking underlying data structure.The stated limitations include non-transparent decision-making and inadequate modeling of rich intra-class variation.
  • DNC approach: DNC assigns test samples to the class whose sub-centroid is nearest in feature space, using representative local means derived from within-class training data.Training alternates between class-wise clustering and representation optimization.
  • DNC approach: DNC couples unsupervised sub-pattern discovery with supervised representation learning, allowing mined local patterns to support classification while classification directly optimizes the representation.This synergy is presented as a core mechanism of the framework.
  • Results: DNC outperforms parametric counterparts by 0.23-0.24% top-1 accuracy on CIFAR-10 and 0.24-0.32% on ImageNet when trained from scratch.These results use ResNet and Swin network architectures.
  • Results: Pixel-wise DNC improves segmentation by 1.6-2.5% mIoU on ADE20K and 1.1-1.9% mIoU on Cityscapes using ImageNet-pretrained nonparametric backbones.The reported segmentation models include FCN, DeepLabV3, and UperNet.
  • Interpretability: Selecting sub-centroids as actual training images provides ad-hoc explainability, with only a 0.12% top-1 accuracy sacrifice while remaining 0.17% better than the parametric counterpart.Human-viewable exemplars anchor the model’s decision-making in concrete observations.

3 DEEP NEAREST CENTROIDS (DNC)

DNC replaces the parametric softmax classifier with distance-based reasoning over class sub-centroids learned from training representations. It directly optimizes the embedding, supports transfer across tasks, and improves transparency and recognition performance.

  • DNC Classifier: DNC assigns each sample to the class whose sub-centroid is nearest in the deep feature space.Each class is represented by multiple local means, allowing complex intra-class variation beyond a unimodal class description.
  • Sub-centroid Estimation: Class sub-centroids are obtained by deterministic within-class clustering of training representations.The clustering uses an optimal-transport relaxation with equipartition constraints to avoid assigning all samples to one cluster.
  • Training: DNC learns visual recognition by directly optimizing the representation rather than a separate parametric classifier.Because sub-centroids derive from data representations, the framework also connects to metric learning by learning a feature-space comparison function.
  • Transferability and Versatility: DNC changes classification decision-making while retaining fixed embedding dimensionality, enabling arbitrary class counts and transfer from ImageNet classification to Cityscapes segmentation.The framework removes the parametric softmax layer and can be applied to image classification and segmentation without slowing inference.
  • Explainability: Restricting sub-centroids to training images gives DNC ad-hoc explainability through inspectable case-based evidence.The sub-centroids are summarized from real observations and are used directly during classification.

4 EXPERIMENT

Experiments evaluate DNC across image classification, semantic segmentation, transfer learning, and human-interpretable predictions. DNC generally improves performance while retaining nonparametric, exemplar-based explanations.

  • Image classification: DNC exceeds parametric classifiers on CIFAR-10 by 0.23% with ResNet50 and 0.24% with ResNet101, using fewer learnable parameters.The backbone architectures and training settings are identical.
  • Image classification: On ImageNet, DNC improves top-1 accuracy by 0.29% and 0.28% over parametric classifiers on ResNet50 and ResNet101.With Swin, DNC reaches 83.26% vs 83.02% on Swin-S and 83.68% vs 83.36% on Swin-B.
  • Transferability: Fine-tuning with DNC-ResNet101 raises DeepLabV3 from 44.1% to 44.6% mIoU on ADE20K and from 78.1% to 78.7% on Cityscapes.DNC-DeepLabV3 further reaches 45.7% and 79.8% mIoU with DNC-ResNet101.
  • Ad-hoc explainability: Anchoring sub-centroids to real training images changes ImageNet top-1 accuracy from 76.49% to 76.37%, while outperforming vanilla ResNet50 at 76.20%.The real-image representatives provide improved interpretability with only marginal degradation from cluster-center sub-centroids.
  • Diagnostic ablation: Increasing class sub-centroids from K=1 to K=4 improves classification from 77.31% to 77.80%, while K=1 to K=10 improves segmentation from 43.2% to 44.3%.Beyond K=10, segmentation gains become marginal or negative.

5 CONCLUSION

The conclusion presents DNC as a simple, nonparametric visual-recognition network that combines within-class structure discovery, transferable representations, and explanations grounded in class exemplars.

  • Conclusion: DNC brings the intuitive Nearest Centroids mechanism into deep neural networks for nonparametric visual recognition.Its classification is based on nonparametric class representatives.
  • Conclusion: Within-class clustering automatically discovers latent data structure through representative sub-centroids.The approach uses sub-pattern mining to represent class distributions.
  • Conclusion: DNC directly supervises representation learning rather than relying on a separate parametric softmax classification layer.The pseudo-code computes image-to-centroid assignments and trains with classification loss.
  • Conclusion: DNC supports transfer across tasks because learned knowledge is stored in a fixed-dimensional embedding rather than class-specific output parameters.The paper describes lossless transferability across visual-recognition tasks.
  • Conclusion: Anchoring class exemplars to real observations gives DNC ad-hoc explainability through human-viewable training examples.The appendix includes additional visual examples of this explainability.
  • Conclusion: The appendix identifies GPU memory cost and broader societal impact as topics for limitations and future work.It also reports directions for future work.

B MORE EXPERIMENTS ON IMAGE CLASSIFICATION

Additional image-classification experiments show that DNC consistently outperforms parametric counterparts across CIFAR-10, CIFAR-100, and ImageNet, including lightweight backbones while using fewer learnable parameters.

  • CIFAR-10: 0.94% higher top-1 accuracy on CIFAR-10, DNC outperforms the parametric counterpart under the same ResNet-18 training setting.The comparison uses the CIFAR-10 test set.
  • CIFAR-100: 0.10% higher top-1 accuracy on ResNet50 and 0.16% higher on ResNet101, DNC consistently outperforms the parametric counterpart on CIFAR-100.CIFAR-100 contains 100 classes, with 500 training and 100 testing images per class.
  • ImageNet: 0.28% higher top-1 accuracy on MobileNet-V2 and 0.30% higher on Swin-Tiny, DNC improves ImageNet performance with lightweight backbones.On MobileNet-V2, DNC reduces learnable parameters from 3.50M to 2.22M.
  • Parameter efficiency: 1.28M fewer learnable parameters on MobileNet-V2, DNC reduces the parameter count from 3.50M to 2.22M.The paper highlights this reduction when the parametric classifier occupies a massive proportion of the lightweight network.

C MORE EXPERIMENTS ON SEMANTIC SEGMENTATION

Additional COCO-Stuff experiments evaluate DNC with FCN, DeepLabV3, and UperNet across ResNet101 and Swin-B backbones under standard training settings.

  • Experimental setup: DNC is evaluated against a parametric counterpart on COCO-Stuff using FCN, DeepLabV3, and UperNet with ResNet101 and Swin-B backbones.COCO-Stuff contains 9K/1K train/test images spanning 80 object classes and 91 stuff classes.
  • Experimental setup: 40K iterations with 512×512 crops and batch size 16 define the standard COCO-Stuff training setting used for all models.The experiments follow the COCO-Stuff training protocol described in the passage.
  • Results: Table 9 reports segmentation mIoU scores on COCO-Stuff for the DNC and parametric counterparts.The supplied passage states that the comparison draws similar conclusions to the earlier segmentation experiments but does not provide the numerical scores.

D ERROR BARS

The paper reports standard-deviation error bars for its main image-classification and semantic-segmentation experiments, based on three training runs with different initialization seeds.

  • Error-bar procedure: Three training runs with different initialization seeds provide the standard-deviation error bars reported for the main experiments.The error bars cover image classification and semantic segmentation.
  • Reported tables: Table 10 reports CIFAR-10 classification top-1 accuracy with error bars.The table concerns the CIFAR-10 test set.
  • Reported tables: Table 11 reports ImageNet validation top-1 and top-5 accuracy with error bars.The table concerns the ImageNet validation set.
  • Reported tables: Table 12 reports segmentation mIoU scores on ADE20K and Cityscapes validation sets with error bars.The table covers the two semantic-segmentation validation datasets.

E EXPERIMENTS ON SUB-CATEGORIES DISCOVERY

Sub-category discovery experiments test whether representations learned from coarse labels capture fine-grained variation. DNC consistently improves or matches the parametric baseline across CIFAR-100 and ImageNet transfer settings.

  • Motivation and mechanism: Within-class clustering represents each class with automatically discovered sub-centroids, capturing multimodal structure, intra-class variance, and sub-class patterns.The passage connects sub-pattern mining with transferable features and algorithmic performance.
  • Evaluation protocol: Coarse-to-fine evaluation measures how well representations learned from coarse-grained labels support fine-grained recognition.The setup assesses discovery of variations within each category.
  • CIFAR-100: 0.12% higher standard top-1 accuracy on both ResNet50 and ResNet101, DNC outperforms the parametric counterpart when transferring from CIFAR-20 to CIFAR-100.The transferred features are evaluated with top-1 nearest-neighbor accuracy in the final feature space.
  • ImageNet: 0.10% and 0.03% improvements on ResNet50 and ResNet101, respectively, DNC improves ImageNet induction accuracy after training on 127 coarse classes.The resulting induction accuracies are 84.39% and 85.91%, respectively.
  • Conclusion: The transfer results suggest that DNC automatically discovers meaningful sub-class patterns not explicitly provided by the supervisory signal.The paper links these latent structures to handling intra-class variance and improving visual recognition.

F TRANSFERABILITY TOWARDS OTHER IMAGE CLASSIFICATION TASK

DNC is evaluated for transfer to CUB-200-2011 using ImageNet-trained representations, with class centers recomputed from the new task’s training data rather than learned as free parameters.

  • CUB-200-2011 contains 11,788 bird photos across 200 categories, with 5,994 training and 5,794 testing images.
  • The CUB-200-2011 experiment uses ResNet50, 100 training epochs, SGD, polynomial learning-rate annealing, and standard flipping, cropping, and normalization.
  • Table 15 reports classification top-1 and top-5 accuracy on the CUB-200-2011 test set.
  • For a new task, DNC fine-tunes only the feature network while recomputing class centers directly from training data using clustering assignments.

G EVALUATION ON IMAGENETV2 TEST SETS

The paper evaluates DNC-ResNet50 on three ImageNetv2 test sets constructed from human selection frequencies, alongside a separate clustering comparison on CIFAR datasets.

  • DNC-ResNet50 is evaluated on ImageNetv2’s Matched Frequency, Threshold0.7, and Top Images test sets.
  • Each ImageNetv2 test set contains ten images per ImageNet class selected using MTurk worker selection frequencies.
  • Matched Frequency samples images according to class-specific selection-frequency distributions, while Threshold0.7 and Top Images use frequency thresholds and rankings.
  • Table 16 reports classification top-1 and top-5 accuracy on the ImageNetv2 test sets.
  • On CIFAR-10 and CIFAR100, Sinkhorn-Knopp clustering performs much better and is more training-efficient than classic k-means within DNC.

I COMPARISON WITH DISTANCE (LEARNING) BASED CLASSIFIERS

DNC is compared with metric- and distance-based classifiers across image classification and segmentation, outperforming the reported counterparts while using compact class representatives for deployment.

  • Image classification: +2.11% on CIFAR-10 and +7.16% on CIFAR-100: DNC significantly outperforms DeepNCM.
  • Image classification: DeepNCM uses one class mean, whereas DNC uses K sub-centers and jointly performs online clustering with supervised representation learning.
  • Image classification: DNC outperforms DeepNCA on ImageNet while avoiding DeepNCA’s need to retain 1.2M training images and compare each test sample against all of them.
  • Deployment: DNC uses four sub-centroids per class for decision-making and adds no extra computation budget during deployment.
  • Semantic segmentation: 0.6% mIoU: DNC surpasses ContrastiveSeg on Cityscapes validation with a ResNet101 backbone and DeepLabV3 architecture.
  • Overall comparison: The three experiments demonstrate DNC effectiveness on image classification and segmentation against other distance-based counterparts.

J ADDITIONAL DIAGNOSTIC EXPERIMENT

Additional diagnostics examine DNC’s output dimensionality, temperature, centroid allocation, memory requirements, interpretability, qualitative segmentation, and relationship to metric-learning literature.

  • Output dimensionality: 76.61% top-1 accuracy: DNC reaches this result on ImageNet validation with final output dimensionality set to 1280.
  • Number of centroids K: +0.06% top-1 accuracy: assigning K according to each class’s training-sample count slightly improves over fixing K = 4 for all classes.
  • External memory: External memory is used only for ImageNet classification because mini-batches may omit many of its 1,000 classes, unlike pixel-wise segmentation batches.
  • Ad-hoc explainability: DNC’s sub-centroid images capture diverse class characteristics, including appearance, viewpoint, scale, and illumination, and can be inspected by users.
  • Ad-hoc explainability: DNC predictions can be interpreted through normalized similarities to corresponding sub-centroids from the top-4 scoring classes.
  • Qualitative segmentation: On qualitative ADE20K and Cityscapes examples, DNC produces more precise segments in challenging scenes involving similar colors, false negatives, and small objects.
  • Additional literature review: Unlike clustering-based self-supervised representation learning, DNC uses class-wise online clustering and sub-centroid classification in supervised learning.

O LIMITATION AND FUTURE WORK

DNC’s main limitations concern training efficiency, out-of-distribution use, and fixed clustering choices. Future work targets faster clustering, adaptive class-specific sub-centroid counts, richer statistics, and further analysis of the method’s properties.

  • Limitation: The Sinkhorn-Knopp algorithm can reduce training efficiency, although three loops per iteration cause only an approximately 5% ImageNet training delay in practice.The authors identify this computational overhead as a direction for future improvement.
  • Limitation: DNC struggles with out-of-distribution data, so its utility in open-world scenarios requires further examination.The paper characterizes this as a common limitation of discriminative classifiers.
  • Future Work: Future work could incorporate more powerful, time-efficient online clustering algorithms to improve training speed and test accuracy.
  • Future Work: Using a fixed number of class centroids K for every class may be suboptimal because intra-class variability differs across classes.Experiments suggest that varying K with class training-sample counts can improve performance.
  • Future Work: The authors propose automatically determining K for different classes and extending DNC beyond first-order statistics to second-order statistics.Second-order statistics may provide more information but introduce computational overhead.
  • Future Work: Further investigation is proposed into intrinsic DNC properties, including robustness against perturbations.
Loading 2209.07383v2…