PaperlayerView exact revision on arXiv

Source-linked AI summary

Visual Recognition with Deep Nearest Centroids

Wenguan Wang, Cheng Han, Tianfei Zhou, Dongfang Liu

arXiv:2209.07383v2cs.CV

TL;DR

Existing visual recognition models rely on opaque parametric classifiers that do not explicitly model latent class structure. DNC instead uses distance-based classification over clustered class sub-centroids while learning only the feature representation, achieving stronger image and pixel recognition with improved transferability and interpretability. Its main scope boundary is that the current centroid count is fixed across classes, although varying it may improve performance.

  • Problem

    Parametric classifiers have abstract, non-transparent parameters, do not directly supervise the representation, and constrain output dimensionality to the number of classes.

    Source: §1 · p. 1, §3 · p. 4

  • Method

    DNC clusters training data within each class into sub-centroids and classifies queries by proximity to these representatives while optimizing the feature representation.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    DNC outperforms parametric counterparts on image classification and improves semantic segmentation across reported datasets, architectures, and models, while enabling exemplar-based interpretability.

    Source: §1 · pp. 2–3, §5 · p. 9

  • Takeaways & Limitations

    DNC provides a simpler, more transparent, transferable, and explainable framework for large-scale visual recognition.

    Source: §1 · p. 2, §5 · p. 9

    The number of class centroids K is fixed identically across classes, despite differing intra-class variability.

    Source: O LIMITATION AND FUTURE WORK · p. 30

Abstract

from arXiv · show

We devise deep nearest centroids (DNC), a conceptually elegant yet surprisingly effective network for large-scale visual recognition, by revisiting Nearest Centroids, one of the most classic and simple classifiers. Current deep models learn the classifier in a fully parametric manner, ignoring the latent data structure and lacking simplicity and explainability. DNC instead conducts nonparametric, case-based reasoning; it utilizes sub-centroids of training samples to describe class distributions and clearly explains the classification as the proximity of test data and the class sub-centroids in the feature space. Due to the distance-based nature, the network output dimensionality is flexible, and all the learnable parameters are only for data embedding. That means all the knowledge learnt for ImageNet classification can be completely transferred for pixel recognition learning, under the "pre-training and fine-tuning" paradigm. Apart from its nested simplicity and intuitive decision-making mechanism, DNC can even possess ad-hoc explainability when the sub-centroids are selected as actual training images that humans can view and inspect. Compared with parametric counterparts, DNC performs better on image classification (CIFAR-10, ImageNet) and greatly boots pixel recognition (ADE20K, Cityscapes), with improved transparency and fewer learnable parameters, using various network architectures (ResNet, Swin) and segmentation models (FCN, DeepLabV3, Swin). We feel this work brings fundamental insights into related fields.

1 INTRODUCTION

DNC revisits Nearest Centroids for visual recognition, replacing opaque parametric classification with distance-based reasoning over learned class sub-centroids. It combines within-class pattern discovery with supervised representation learning and reports stronger performance, transferability, and interpretability across recognition tasks.

Motivation

  • Parametric softmax classifiers use abstract class-specific parameters, making their decisions difficult to explain and potentially overlooking underlying data structure.The stated limitations include non-transparent decision-making and inadequate modeling of rich intra-class variation.Source: §1 · p. 1, Figure 1 · p. 2

DNC approach

  • DNC assigns test samples to the class whose sub-centroid is nearest in feature space, using representative local means derived from within-class training data.Training alternates between class-wise clustering and representation optimization.Source: §1 · pp. 1–2
  • DNC couples unsupervised sub-pattern discovery with supervised representation learning, allowing mined local patterns to support classification while classification directly optimizes the representation.This synergy is presented as a core mechanism of the framework.Source: §1 · p. 2, Figure 2 · p. 4

Results

  • DNC outperforms parametric counterparts by 0.23-0.24% top-1 accuracy on CIFAR-10 and 0.24-0.32% on ImageNet when trained from scratch.These results use ResNet and Swin network architectures.Source: §1 · pp. 2–3
  • Pixel-wise DNC improves segmentation by 1.6-2.5% mIoU on ADE20K and 1.1-1.9% mIoU on Cityscapes using ImageNet-pretrained nonparametric backbones.The reported segmentation models include FCN, DeepLabV3, and UperNet.Source: §1 · pp. 2–3

Interpretability

  • Selecting sub-centroids as actual training images provides ad-hoc explainability, with only a 0.12% top-1 accuracy sacrifice while remaining 0.17% better than the parametric counterpart.Human-viewable exemplars anchor the model’s decision-making in concrete observations.Source: §1 · pp. 2–3

3 DEEP NEAREST CENTROIDS (DNC)

DNC replaces the parametric softmax classifier with distance-based reasoning over class sub-centroids learned from training representations. It directly optimizes the embedding, supports transfer across tasks, and improves transparency and recognition performance.

DNC Classifier

  • DNC assigns each sample to the class whose sub-centroid is nearest in the deep feature space.Each class is represented by multiple local means, allowing complex intra-class variation beyond a unimodal class description.Source: §3 · p. 4

Sub-centroid Estimation

  • Class sub-centroids are obtained by deterministic within-class clustering of training representations.The clustering uses an optimal-transport relaxation with equipartition constraints to avoid assigning all samples to one cluster.Source: §3 · p. 5

Training

  • DNC learns visual recognition by directly optimizing the representation rather than a separate parametric classifier.Because sub-centroids derive from data representations, the framework also connects to metric learning by learning a feature-space comparison function.Source: §3 · p. 5

Transferability and Versatility

  • DNC changes classification decision-making while retaining fixed embedding dimensionality, enabling arbitrary class counts and transfer from ImageNet classification to Cityscapes segmentation.The framework removes the parametric softmax layer and can be applied to image classification and segmentation without slowing inference.Source: §3 · pp. 4, 6

Explainability

  • Restricting sub-centroids to training images gives DNC ad-hoc explainability through inspectable case-based evidence.The sub-centroids are summarized from real observations and are used directly during classification.Source: §3 · pp. 5–6

4 EXPERIMENT

Experiments evaluate DNC across image classification, semantic segmentation, transfer learning, and human-interpretable predictions. DNC generally improves performance while retaining nonparametric, exemplar-based explanations.

Image classification

  • DNC exceeds parametric classifiers on CIFAR-10 by 0.23% with ResNet50 and 0.24% with ResNet101, using fewer learnable parameters.The backbone architectures and training settings are identical.Source: §4.1 · p. 6
  • On ImageNet, DNC improves top-1 accuracy by 0.29% and 0.28% over parametric classifiers on ResNet50 and ResNet101.With Swin, DNC reaches 83.26% vs 83.02% on Swin-S and 83.68% vs 83.36% on Swin-B.Source: §4.1 · p. 7

Transferability

  • Fine-tuning with DNC-ResNet101 raises DeepLabV3 from 44.1% to 44.6% mIoU on ADE20K and from 78.1% to 78.7% on Cityscapes.DNC-DeepLabV3 further reaches 45.7% and 79.8% mIoU with DNC-ResNet101.Source: §4.2 · p. 8

Ad-hoc explainability

  • Anchoring sub-centroids to real training images changes ImageNet top-1 accuracy from 76.49% to 76.37%, while outperforming vanilla ResNet50 at 76.20%.The real-image representatives provide improved interpretability with only marginal degradation from cluster-center sub-centroids.Source: §4.3 · p. 8

Diagnostic ablation

  • Increasing class sub-centroids from K=1 to K=4 improves classification from 77.31% to 77.80%, while K=1 to K=10 improves segmentation from 43.2% to 44.3%.Beyond K=10, segmentation gains become marginal or negative.Source: §4.4 · p. 9

5 CONCLUSION

The conclusion presents DNC as a simple, nonparametric visual-recognition network that combines within-class structure discovery, transferable representations, and explanations grounded in class exemplars.

Conclusion

  • DNC brings the intuitive Nearest Centroids mechanism into deep neural networks for nonparametric visual recognition.Its classification is based on nonparametric class representatives.Source: §5 · p. 9
  • Within-class clustering automatically discovers latent data structure through representative sub-centroids.The approach uses sub-pattern mining to represent class distributions.Source: §5 · p. 9, A PSEUDO CODE OF DNC AND CODE RELEASE · p. 17
  • DNC directly supervises representation learning rather than relying on a separate parametric softmax classification layer.The pseudo-code computes image-to-centroid assignments and trains with classification loss.Source: §5 · p. 9, A PSEUDO CODE OF DNC AND CODE RELEASE · p. 17
  • DNC supports transfer across tasks because learned knowledge is stored in a fixed-dimensional embedding rather than class-specific output parameters.The paper describes lossless transferability across visual-recognition tasks.Source: §5 · p. 9, SUMMARY OF THE APPENDIX · p. 16
  • Anchoring class exemplars to real observations gives DNC ad-hoc explainability through human-viewable training examples.The appendix includes additional visual examples of this explainability.Source: §5 · p. 9, SUMMARY OF THE APPENDIX · p. 16
  • The appendix identifies GPU memory cost and broader societal impact as topics for limitations and future work.It also reports directions for future work.Source: SUMMARY OF THE APPENDIX · p. 16

B MORE EXPERIMENTS ON IMAGE CLASSIFICATION

Additional image-classification experiments show that DNC consistently outperforms parametric counterparts across CIFAR-10, CIFAR-100, and ImageNet, including lightweight backbones while using fewer learnable parameters.

CIFAR-10

  • 0.94% higher top-1 accuracy on CIFAR-10, DNC outperforms the parametric counterpart under the same ResNet-18 training setting.The comparison uses the CIFAR-10 test set.Source: B MORE EXPERIMENTS ON IMAGE… · p. 18

CIFAR-100

  • 0.10% higher top-1 accuracy on ResNet50 and 0.16% higher on ResNet101, DNC consistently outperforms the parametric counterpart on CIFAR-100.CIFAR-100 contains 100 classes, with 500 training and 100 testing images per class.Source: B MORE EXPERIMENTS ON IMAGE… · p. 18

ImageNet

  • 0.28% higher top-1 accuracy on MobileNet-V2 and 0.30% higher on Swin-Tiny, DNC improves ImageNet performance with lightweight backbones.On MobileNet-V2, DNC reduces learnable parameters from 3.50M to 2.22M.Source: B MORE EXPERIMENTS ON IMAGE… · p. 18

Parameter efficiency

  • 1.28M fewer learnable parameters on MobileNet-V2, DNC reduces the parameter count from 3.50M to 2.22M.The paper highlights this reduction when the parametric classifier occupies a massive proportion of the lightweight network.Source: B MORE EXPERIMENTS ON IMAGE… · p. 18

C MORE EXPERIMENTS ON SEMANTIC SEGMENTATION

Additional COCO-Stuff experiments evaluate DNC with FCN, DeepLabV3, and UperNet across ResNet101 and Swin-B backbones under standard training settings.

Experimental setup

  • DNC is evaluated against a parametric counterpart on COCO-Stuff using FCN, DeepLabV3, and UperNet with ResNet101 and Swin-B backbones.COCO-Stuff contains 9K/1K train/test images spanning 80 object classes and 91 stuff classes.Source: C MORE EXPERIMENTS ON SEMANTIC… · p. 19
  • 40K iterations with 512×512 crops and batch size 16 define the standard COCO-Stuff training setting used for all models.The experiments follow the COCO-Stuff training protocol described in the passage.Source: C MORE EXPERIMENTS ON SEMANTIC… · p. 19

Results

  • Table 9 reports segmentation mIoU scores on COCO-Stuff for the DNC and parametric counterparts.The supplied passage states that the comparison draws similar conclusions to the earlier segmentation experiments but does not provide the numerical scores.Source: Table 9 · p. 19, C MORE EXPERIMENTS ON SEMANTIC… · p. 19

D ERROR BARS

The paper reports standard-deviation error bars for its main image-classification and semantic-segmentation experiments, based on three training runs with different initialization seeds.

Error-bar procedure

  • Three training runs with different initialization seeds provide the standard-deviation error bars reported for the main experiments.The error bars cover image classification and semantic segmentation.Source: D ERROR BARS · p. 19

Reported tables

  • Table 10 reports CIFAR-10 classification top-1 accuracy with error bars.The table concerns the CIFAR-10 test set.Source: Table 10 · p. 19
  • Table 11 reports ImageNet validation top-1 and top-5 accuracy with error bars.The table concerns the ImageNet validation set.Source: Table 11 · p. 19
  • Table 12 reports segmentation mIoU scores on ADE20K and Cityscapes validation sets with error bars.The table covers the two semantic-segmentation validation datasets.Source: Table 12 · p. 20

E EXPERIMENTS ON SUB-CATEGORIES DISCOVERY

Sub-category discovery experiments test whether representations learned from coarse labels capture fine-grained variation. DNC consistently improves or matches the parametric baseline across CIFAR-100 and ImageNet transfer settings.

Motivation and mechanism

  • Within-class clustering represents each class with automatically discovered sub-centroids, capturing multimodal structure, intra-class variance, and sub-class patterns.The passage connects sub-pattern mining with transferable features and algorithmic performance.Source: E EXPERIMENTS ON SUB-CATEGORIES… · p. 20

Evaluation protocol

  • Coarse-to-fine evaluation measures how well representations learned from coarse-grained labels support fine-grained recognition.The setup assesses discovery of variations within each category.Source: E EXPERIMENTS ON SUB-CATEGORIES… · p. 20

CIFAR-100

  • 0.12% higher standard top-1 accuracy on both ResNet50 and ResNet101, DNC outperforms the parametric counterpart when transferring from CIFAR-20 to CIFAR-100.The transferred features are evaluated with top-1 nearest-neighbor accuracy in the final feature space.Source: E EXPERIMENTS ON SUB-CATEGORIES… · p. 20, Table 13 · p. 21

ImageNet

Conclusion

  • The transfer results suggest that DNC automatically discovers meaningful sub-class patterns not explicitly provided by the supervisory signal.The paper links these latent structures to handling intra-class variance and improving visual recognition.Source: E EXPERIMENTS ON SUB-CATEGORIES… · p. 21

F TRANSFERABILITY TOWARDS OTHER IMAGE CLASSIFICATION TASK

DNC is evaluated for transfer to CUB-200-2011 using ImageNet-trained representations, with class centers recomputed from the new task’s training data rather than learned as free parameters.

G EVALUATION ON IMAGENETV2 TEST SETS

The paper evaluates DNC-ResNet50 on three ImageNetv2 test sets constructed from human selection frequencies, alongside a separate clustering comparison on CIFAR datasets.

I COMPARISON WITH DISTANCE (LEARNING) BASED CLASSIFIERS

DNC is compared with metric- and distance-based classifiers across image classification and segmentation, outperforming the reported counterparts while using compact class representatives for deployment.

Image classification

Deployment

Semantic segmentation

Overall comparison

J ADDITIONAL DIAGNOSTIC EXPERIMENT

Additional diagnostics examine DNC’s output dimensionality, temperature, centroid allocation, memory requirements, interpretability, qualitative segmentation, and relationship to metric-learning literature.

Output dimensionality

Number of centroids K

External memory

Ad-hoc explainability

Qualitative segmentation

Additional literature review

O LIMITATION AND FUTURE WORK

DNC’s main limitations concern training efficiency, out-of-distribution use, and fixed clustering choices. Future work targets faster clustering, adaptive class-specific sub-centroid counts, richer statistics, and further analysis of the method’s properties.

Limitation

  • The Sinkhorn-Knopp algorithm can reduce training efficiency, although three loops per iteration cause only an approximately 5% ImageNet training delay in practice.The authors identify this computational overhead as a direction for future improvement.Source: O LIMITATION AND FUTURE WORK · p. 29
  • DNC struggles with out-of-distribution data, so its utility in open-world scenarios requires further examination.The paper characterizes this as a common limitation of discriminative classifiers.Source: O LIMITATION AND FUTURE WORK · pp. 29–30

Future Work

  • Future work could incorporate more powerful, time-efficient online clustering algorithms to improve training speed and test accuracy.Source: O LIMITATION AND FUTURE WORK · p. 30
  • Using a fixed number of class centroids K for every class may be suboptimal because intra-class variability differs across classes.Experiments suggest that varying K with class training-sample counts can improve performance.Source: O LIMITATION AND FUTURE WORK · p. 30
  • The authors propose automatically determining K for different classes and extending DNC beyond first-order statistics to second-order statistics.Second-order statistics may provide more information but introduce computational overhead.Source: O LIMITATION AND FUTURE WORK · p. 30
  • Further investigation is proposed into intrinsic DNC properties, including robustness against perturbations.Source: O LIMITATION AND FUTURE WORK · p. 30

Related papers

Loading 2209.07383v2…