PaperlayerView exact revision on arXiv

Source-linked AI summary

Hyperspherical Prototype Networks

Pascal Mettes, Elise van der Pol, Cees G. M. Snoek

arXiv:1901.10514v3cs.LGstat.ML

TL;DR

The paper addresses limitations of learned, fixed-size output spaces and data-dependent class prototypes. It introduces a priori optimized hyperspherical prototypes for classification and regression, finding benefits across classification, regression, and joint multi-task settings over the compared approaches.

  • Problem

    Existing classification and regression methods use fixed-size, fully parametric output spaces, while mean-based prototypes require continual re-estimation during training.

  • Method

    Hyperspherical prototype networks place prototypes a priori with large-margin separation, extend them to regression through two bounds, and optimize both tasks with a shared cosine-similarity loss.

  • Results

    The experiments show benefits for classification, regression, and multi-task problems over prototype alternatives and standard loss-based approaches.

  • Takeaways & Limitations

    The approach supports arbitrary output dimensionality and joint classification-regression training in a shared output space without task-weighting hyperparameters.

Abstract

from arXiv · show

This paper introduces hyperspherical prototype networks, which unify classification and regression with prototypes on hyperspherical output spaces. For classification, a common approach is to define prototypes as the mean output vector over training examples per class. Here, we propose to use hyperspheres as output spaces, with class prototypes defined a priori with large margin separation. We position prototypes through data-independent optimization, with an extension to incorporate priors from class semantics. By doing so, we do not require any prototype updating, we can handle any training size, and the output dimensionality is no longer constrained to the number of classes. Furthermore, we generalize to regression, by optimizing outputs as an interpolation between two prototypes on the hypersphere. Since both tasks are now defined by the same loss function, they can be jointly trained for multi-task problems. Experimentally, we show the benefit of hyperspherical prototype networks for classification, regression, and their combination over other prototype methods, softmax cross-entropy, and mean squared error approaches.

1 Introduction

The paper replaces learned, fixed-size output spaces with a priori hyperspherical spaces that encode inductive biases and support both classification and regression. Fixed prototypes, semantic organization, and a shared optimization framework enable flexible outputs and joint training without further tuning.

  • Standard classification and regression learn fully parametric output spaces with fixed dimensionality, limiting the incorporation of inductive biases such as margin separation and source-data knowledge.
  • Unlike mean-based prototypes that require continual re-estimation or few-shot approximations, the proposed prototypes are fixed independently of training data.
  • Hyperspherical prototype networks define output spaces a priori, uniformly separating classification prototypes and using prototype bounds for smooth regression transitions.
  • Classification prototypes are optimized for approximate uniformity and can incorporate privileged information to encode semantic class structures.
  • A shared cosine-similarity-based framework supports classification and regression in the same output space and enables joint optimization without further tuning.

2 Hyperspherical prototypes

Hyperspherical prototype networks predefine class prototypes on hyperspherical output spaces, then train predictions toward them using angular similarity. The same framework extends to regression and jointly supports classification and regression with a shared cosine-similarity loss.

  • 2.1 Classification: Classification minimizes the angle between each network output and the prototype of its ground-truth class.The resulting loss maximizes cosine similarity, while prediction selects the class prototype with the highest cosine similarity.
  • 2 Hyperspherical prototypes: Class prototypes are positioned before training as approximately uniform points on a hypersphere, incorporating large-margin separation and simplicity.Uniform placement is approximated through differentiable optimization because the general spherical-code problem lacks exact solutions.
  • 2.1 Classification: Prototype optimization uses matrix-form pairwise similarities, updates prototypes iteratively, and re-projects them onto the hypersphere with ℓ2 normalization.Subtracting the identity avoids self-selection when minimizing each prototype’s nearest cosine similarity.
  • 2.1 Classification: Semantic class priors are incorporated by ranking prototype relations to match the similarity ordering of class word embeddings while retaining separation.The ranking loss is combined with the prototype-separation objective rather than directly aligning raw similarities.
  • 2.2 Regression: Regression uses upper and lower hyperspherical prototypes, with outputs interpolating between them through cosine similarity to the bounds.The squared loss compares normalized ground-truth values with measured cosine similarity to the upper-bound prototype; higher-dimensional outputs provide additional optimization degrees of freedom.
  • 2.3 Joint regression and classification: Classification and regression share a cosine-similarity loss and can be trained in one output space by summing their losses without task-balancing hyperparameters.Regression bounds occupy opposite directions along one axis, while remaining axes can separate classification prototypes.

3 Experiments

Experiments show that hyperspherical prototypes support classification with flexible output dimensions, benefit from privileged information, and outperform prototype and softmax baselines in several settings. They also provide strong regression results and enable joint regression-classification training in a shared output space.

  • Classification: Hyperspherical prototypes maintain classification accuracy while removing up to three quarters of the output space and outperform word2vec prototypes by 7.4 percentage points on CIFAR-100.The approach also handles output dimensionalities unavailable to one-hot prototypes.
  • Classification: The approach achieves higher mean and maximum prototype separation than one-hot and word2vec prototypes, while word2vec has low minimum separation for semantically related classes.These separation limitations are reflected in classification results.
  • Classification: With five output dimensions, privileged information improves CIFAR-100 accuracy from 28.7 ± 0.4 to 37.0 ± 0.8.The benefit is especially pronounced for small or mismatched output spaces and is associated with faster convergence.
  • Classification: Against Deep NCM, hyperspherical prototypes converge faster, reach better results, vary more smoothly, and avoid repeated prototype updates.The comparison uses the same network architecture and hyperparameter settings.
  • Classification: When examples per class are scarce or uneven, hyperspherical prototypes are preferred to softmax cross-entropy; on one such setting, accuracy is 47.3 ± 0.1 versus 43.0 ± 0.6.On CIFAR-100 with many evenly distributed examples, the methods obtain similar scores.
  • Regression: For creation-year regression, hyperspherical prototypes achieve lower mean absolute error than the baseline at learning rates 1e-2 and 1e-3 for S1 and S2 outputs.At learning rate 1e-3, errors are 76.3 and 73.2 versus 110.0 and 109.9; at 1e-2, they are 84.4 and 82.9 versus 210.7 and 339.9.
  • Joint regression and classification: On OmniArt, the shared hyperspherical output space outperforms the multi-task baseline across regression-classification weight settings.The joint task also lowers creation-year error relative to the separate regression experiment.

4 Related work

Prior prototype networks define class prototypes from training-example means, creating continuously moving targets during training. Hyperspherical prototype networks instead fix highly separated prototypes before learning and extend them to regression.

  • Prototype-based networks assign examples to class prototypes defined as mean vectors, with variants using multiple prototypes per class.
  • Because network outputs change during training, mean-based prototype locations also change, requiring expensive dataset-wide re-estimation or approximation.
  • The proposed method structures the output space before training, placing hyperspherical prototypes with large margins and optimizing cosine distance to assigned prototypes.
  • Unlike related methods that use predefined Euclidean prototypes for metric learning, this approach uses hyperspherical prototypes for classification and regression in deep networks.
  • Angular-margin and hypersphere-normalized softmax methods optimize class separation during training, whereas this method fixes highly separated prototypes beforehand.

5 Conclusions

The paper proposes hyperspherical prototype networks that define well-separated prototypes independently of training data for classification and regression. Experiments report effective, fast-to-train, flexible networks that operate in compact output spaces.

  • Hyperspherical prototype networks avoid the chicken-or-egg problem by distributing class prototypes with large margins before training.
  • The framework supports class-semantic priors, continuous outputs, and joint classification-regression optimization in one output space.
  • The authors report that hyperspherical prototypes are effective, fast to train, and easy to implement across classification and regression tasks.
  • Potential future directions include incremental learning and open set learning.

A Evaluating hyperspherical prototypes on DenseNet-121

DenseNet-121 experiments reproduce the main conclusions: hyperspherical prototypes reduce output dimensionality while maintaining performance and outperform baseline prototypes on both datasets.

  • Hyperspherical prototypes enable a large reduction in output dimensionality while maintaining performance with DenseNet-121.
  • The method outperforms baseline prototypes on both evaluated datasets.
  • Unlike one-hot encodings, hyperspherical prototypes can use any output dimensionality and obtain the best scores across datasets.

B Accuracy per epoch for prototypes with privileged information

On CIFAR-100 with ResNet-32, privileged class-semantic information improves classification accuracy and convergence speed, with the strongest benefit in small output spaces.

  • The experiment compares models with and without privileged information using 3, 5, 10, and 25 output dimensions.
  • Privileged information produces better classification results and faster convergence across training epochs.
  • The benefit of privileged information is especially pronounced when the output space is small.
Loading 1901.10514v3…