PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Hyperspherical Learning

Weiyang Liu, Yan-Ming Zhang, Xingguo Li, Zhiding Yu, Bo Dai, Tuo Zhao, Le Song

arXiv:1711.03189v5cs.LGcs.CVstat.ML

TL;DR

Deep CNNs become increasingly difficult to train as depth and parameter space grow. The paper introduces hyperspherical convolution and angular supervision, and reports faster convergence with comparable or better classification accuracy than conventional convolutional networks.

  • Problem

    Deep CNNs achieve strong visual representations but face training difficulties that worsen with increasing depth and parameter space.

  • Method

    SphereNet replaces inner-product convolution with SphereConv on unit hyperspheres and uses generalized angular softmax for angular supervision.

  • Results

    SphereNet alleviates training difficulty through easier optimization and faster convergence while achieving comparable or better classification accuracy than convolutional counterparts.

  • Takeaways & Limitations

    Hyperspherical learning provides an alternative convolutional framework that extends angular representations, supervision, and normalization across network layers.

  • Takeaways & Limitations

    SphereNet’s performance gains depend on sufficient network width, while each neuron is slightly more computationally expensive and SphereConv operators are mostly prefixed.

Abstract

from arXiv · show

Convolution as inner product has been the founding basis of convolutional neural networks (CNNs) and the key to end-to-end visual representation learning. Benefiting from deeper architectures, recent CNNs have demonstrated increasingly strong representation abilities. Despite such improvement, the increased depth and larger parameter space have also led to challenges in properly training a network. In light of such challenges, we propose hyperspherical convolution (SphereConv), a novel learning framework that gives angular representations on hyperspheres. We introduce SphereNet, deep hyperspherical convolution networks that are distinct from conventional inner product based convolutional networks. In particular, SphereNet adopts SphereConv as its basic convolution operator and is supervised by generalized angular softmax loss - a natural loss formulation under SphereConv. We show that SphereNet can effectively encode discriminative representation and alleviate training difficulty, leading to easier optimization, faster convergence and comparable (even better) classification accuracy over convolutional counterparts. We also provide some theoretical insights for the advantages of learning on hyperspheres. In addition, we introduce the learnable SphereConv, i.e., a natural improvement over prefixed SphereConv, and SphereNorm, i.e., hyperspherical learning as a normalization method. Experiments have verified our conclusions.

1 Introduction

Deep CNNs offer strong representations but become harder to train as depth and parameter count increase. SphereNet addresses this by replacing inner-product convolutions with hyperspherical, angle-based learning and related angular supervision.

  • Deep CNNs face overfitting, vanishing or exploding gradients, covariate shift, and greater training difficulty as architectures deepen.
  • SphereNet projects parameter learning onto unit hyperspheres, making activations depend on angular distance rather than inner products.
  • SphereConv forms the network’s basic operator, while generalized angular softmax supervises activations from the final layer.
  • Angular learning is motivated by training stability, efficiency, bounded activations, and the view that phase-like information preserves discriminative image structure.
  • The framework extends hyperspherical constraints across layers and introduces linear, cosine, sigmoid, learnable SphereConv, and SphereNorm variants.
  • SphereNet can replace convolution operators and loss functions in GoogLeNet, VGG, and ResNet, questioning whether inner products are optimal correlation measures for every task.

2 Hyperspherical Convolutional Operator

SphereConv replaces inner-product convolution with angular similarity on unit hyperspheres, using functions of the geodesic angle between kernels and local patches. The framework includes several operator forms, theoretical conditioning benefits, and extensions for regularization, normalization, and learnable curvature.

  • 2.1 Definition: SphereConv computes a function of the angle between a kernel and local patch, treating the angle as geodesic distance on a unit hypersphere.Its general form is F_s(w, x) = g(θ(w,x)) + b_Fs, with outputs constrained to [−1, 1] for convenient computation.
  • 2.1 Definition: The framework instantiates SphereConv with linear, cosine, and sigmoid operators, where sigmoid curvature controls behavior from step-like to approximately linear.The sigmoid parameter k can later be learned by back-propagation rather than fixed in advance.
  • 2.3 Theoretical Insights: Scaling can worsen conventional matrix-factorization conditioning by Ω(c^2), whereas cosine SphereConv preserves Hessian eigenvalues and conditioning under reciprocal parameter scaling.The analysis attributes the conventional instability to unbalanced global optima and states that SphereConv eliminates this scaling-induced conditioning issue.
  • 2.4 Discussion: SphereConv is an alternative correlation measure to inner-product convolution, while normalized convolution is equivalent to cosine SphereConv.The distinction is framed as using different metric spaces: conventional convolution uses inner products, whereas SphereConv uses angular functions.
  • 2.4 Discussion: SphereNet regularizes kernels through orthogonality instead of ℓ2 weight decay, and SphereConv can also provide normalization distinct from batch normalization.SphereNorm normalizes using feature-map spatial information and kernel weights rather than mini-batch statistics; learnable SphereConv updates sigmoid curvature by back-propagation.

3 Learning Objective on Hyperspheres

The paper formulates classification on hyperspheres by normalizing weights and removing biases so decisions depend on angular relationships, then extends this objective with angular margins. It also discusses optimization trade-offs among different angular losses, including instability and difficult training for some choices.

  • Weight-normalized softmax makes class decisions depend on angles by normalizing weights and setting biases to zero, with optional biases for imbalanced data.
  • Generalized angular softmax extends weight-normalized softmax with a multiplicative margin parameter m that favors larger angular margins.
  • The angular score function g(·) can use linear, cosine, or sigmoid forms, producing corresponding weight-normalized losses.
  • A-Softmax is the cosine generalized angular softmax special case, while weight-normalized softmax corresponds to m = 1.
  • Some angular losses can introduce saddle points or unstable optimization, whereas linear generalized angular softmax may yield more stable training; sphere-normalized softmax was difficult to optimize from scratch.

4 Experiments and Results

Experiments evaluate SphereNet through ablations, convergence studies, normalization tests, and image-classification benchmarks. Across these settings, SphereNet generally improves convergence stability or accuracy over conventional CNN baselines, with strongest gains in deeper and wider networks.

  • Loss and operator ablations: SphereConv operators consistently outperform conventional convolution, and large-margin losses such as A-Softmax and GA-Softmax further boost accuracy.Table 1 reports classification accuracy under different operators and losses.
  • Width ablation: Wider SphereNets surpass CNN baselines with faster and more stable convergence, whereas narrow models perform similarly or slightly worse.The width comparison varies filters across convolutional stages.
  • Nonlinearity ablation: Without ReLU, SphereNet still significantly outperforms CNN, indicating that SphereConv supplies substantial nonlinearity and representation capacity.Both methods use 18-layer CNNs with BatchNorm in this comparison.
  • Convergence: SphereResNet converges faster than ResNet and achieves higher final accuracy, while a 69-layer plain SphereNet reaches nearly 90% accuracy without residual shortcuts.The convergence studies cover CIFAR-10/CIFAR-10+ and ultra-deep plain networks.
  • Learnable SphereConv: Learnable sigmoid SphereConv reaches 91.64% on CIFAR-10, exceeding the best fixed sigmoid SphereConv result of 91.22%.Both results use cosine W-Softmax and the same 9-layer architecture without full data augmentation.
  • SphereNorm: SphereNorm converges faster and more stably than BatchNorm at similar final accuracy, while combining both yields the fastest convergence and highest final accuracy.The evaluation uses small mini-batches and a 9-layer CNN on CIFAR-10.
  • Image classification: On CIFAR-10+ and CIFAR-100, SphereNet outperforms many state-of-the-art methods and is comparable to the much deeper ResNet-1001.The study evaluates several SphereConv and hyperspherical-loss combinations with ResNet-32 as the baseline architecture.
  • Image classification: On ImageNet-2012, both SphereResNet-18 variants converge faster than ResNet-18; v2 also achieves slightly better accuracy, while v1 converges fastest with comparable accuracy.The two variants differ in whether shortcut 1×1 convolutions use SphereConv.

5 Limitations and Future Work

The authors identify limitations involving network width, computational complexity, and prefixed SphereConvs, while proposing corresponding future work.

  • SphereNets provide larger gains when networks are wide enough; narrower networks converge faster but may yield slightly worse, still comparable, recognition accuracy.
  • Each SphereConv neuron has slightly higher computational complexity than a conventional CNN neuron.
  • Most SphereConvs remain prefixed, motivating learnable operators, more efficient angle computation, and improved angular regularization as future work.

A Network Architectures

The appendix describes plain CNN and ResNet architectures using different convolutional layers and specifies their unit and downsampling organization.

  • Plain CNN architectures are organized into Conv1.x, Conv2.x, and Conv3.x units that may contain multiple convolution layers.For example, [3×3, 64]×3 denotes three cascaded 3×3 convolution layers with 64 filters.
  • ResNet architectures use Conv0.x through Conv4.x units, with residual units shown in double-column brackets.These networks are essentially based on the cited ResNet design, with possible differences in filters per layer.
  • Downsampling in the ResNet architectures is performed by convolutions with stride 2.

B Experimental Details for Imagenet-2012

The ImageNet-2012 experiments use minimal augmentation and specify distinct SphereConv and loss configurations for the two SphereResNet-18 variants.

  • ImageNet inputs are resized to 256 × 256, randomly cropped to 224 × 224, and randomly flipped horizontally.
  • SphereResNet-18-v1 uses cosine SphereConv with cosine W-Softmax loss, whereas v2 uses cosine SphereConv with standard softmax loss.

C More Discussions for Sphere-normalized Softmax Loss

Sphere-normalized softmax applies SphereConv to the final fully connected layer, then rescales its logits to make optimization practical.

  • S-Softmax applies SphereConv to the fully connected layer within the softmax loss.
  • Because direct SphereConv logits are difficult to converge, a scaling factor s expands their range from [−1, 1] to [−s, s].Values of s from 10 to 70 generally work well in practice.

D.1 Proof of Lemma 1

The proof analyzes the Hessian at globally optimal factorizations and shows how scaling the factors affects its spectrum. Orthogonal-factorization invariance creates zero eigenvalues, while scaling changes the restricted conditioning.

  • D.1 Proof of Lemma 1: Scaling the factors by c changes the Hessian trace through c^2-weighted Gram terms.The trace expression contains contributions from V^⊤V ⊗ I_n and U^⊤U ⊗ I_m, with scaling by c^2.
  • D.1 Proof of Lemma 1: At a global optimum, UV^⊤ = F, and orthogonal transformations preserve the represented product while producing r(r−1)/2 zero Hessian eigenvalues.The invariance is UV^⊤ = UR(VR)^⊤ for any orthogonal R.
  • D.1 Proof of Lemma 1: The largest Hessian eigenvalue grows on the order of Θ(c^2), while the smallest nonzero eigenvalue does not increase under the same scaling.The proof uses perturbation bounds and assumes U and V are balanced.

D.2 Proof of Lemma 2

The proof derives gradients and Hessians for the normalized objective, then shows that entry-wise normalization preserves the entire Hessian spectrum under the stated normalization operation.

  • D.2 Proof of Lemma 2: After each SGD iteration, entry-wise normalization sets D_U = I_n and D_V = I_m before evaluating the gradient.The normalized gradient is derived after applying this post-iteration normalization to U and V.
  • D.2 Proof of Lemma 2: The Hessian is constructed from Hadamard-product blocks involving 2UV^⊤ − F and pairwise coordinate products of U’s columns.The displayed block structure indexes combinations of U_{:,i} ⊙ U_{:,j}.
  • D.2 Proof of Lemma 2: λ_i(∇^2G_S(eU,eV)) = λ_i(∇^2G_S(U,V)) for every i, so normalization preserves all Hessian eigenvalues.The equality follows from normalizing each row of U and V by its ℓ_2 norm.
Loading 1711.03189v5…