PaperlayerView exact revision on arXiv

Source-linked AI summary

Understanding Locally Competitive Networks

Rupesh Kumar Srivastava, Jonathan Masci, Faustino Gomez, Jürgen Schmidhuber

arXiv:1410.1165v3cs.NEcs.LG

TL;DR

Locally competitive activation functions have been associated with improved training, but their behavior and advantages over sigmoidal networks remain incompletely understood. This paper analyzes how training organizes their subnetworks and evaluates activation patterns as retrieval codes. The authors report subnetwork structure consistent with their account and promising retrieval results across datasets of varying size and complexity.

  • Problem

    The behavior and advantages of locally competitive networks over sigmoidal networks, especially during learning, remain incompletely understood.

    Source: §1 · p. 1

  • Method

    The paper analyzes activated subnetworks as models that switch among submodels and evaluates their binary activation codes using nearest-neighbor classification.

    Source: §1 · p. 1, §3 · p. 3, §3.3 · p. 5

  • Results

    Subnetwork patterns in held-out examples are consistent with the switching-submodels hypothesis, and their encodings yield promising retrieval results on MNIST, CIFAR-10, CIFAR-100, and ImageNet.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    Useful binary descriptors can be obtained directly from a classification-trained neural network without additional training, and the authors connect locally competitive architectures with competitive learning.

    Source: §5 · p. 8

    The ImageNet comparison was not exhaustive, and the authors did not investigate more sophisticated hashing techniques.

    Source: §4.2 · p. 7

Abstract

from arXiv · show

Recently proposed neural network activation functions such as rectified linear, maxout, and local winner-take-all have allowed for faster and more effective training of deep neural architectures on large and complex datasets. The common trait among these functions is that they implement local competition between small groups of computational units within a layer, so that only part of the network is activated for any given input pattern. In this paper, we attempt to visualize and understand this self-modularization, and suggest a unified explanation for the beneficial properties of such networks. We also show how our insights can be directly useful for efficiently performing retrieval over large datasets using neural networks.

1 INTRODUCTION

The paper examines why locally competitive activation functions can outperform sigmoidal networks, interpreting them as models that switch among subnetworks. It analyzes subnetworks across examples and uses their activation patterns for retrieval.

1 INTRODUCTION

  • ReLU, maxout, and LWTA use local competition, while prior work reports faster, better training and beneficial properties compared with sigmoidal networks.Reported properties include avoiding unsupervised weight initialization, better gradient flow, and mitigation of catastrophic forgetting.Source: §1 · p. 1
  • The paper interprets locally competitive networks as models that switch among subnetworks, with similar subnetworks responding to similar patterns.Analysis of held-out examples finds subnetwork structure consistent with this hypothesis, which the authors use to explain improved credit assignment.Source: §1 · p. 1
  • Different input patterns activate different subsets of units, illustrating the subnetworks that motivate the paper’s model-of-models interpretation.Figure 2 depicts subnetworks for 100 examples and 10 ReLUs; unit 3 is inactive for all examples.Source: §1 · p. 1, Figure 2 · p. 2
  • A simple encoding of activated units can represent examples for retrieval, with promising results reported on MNIST, CIFAR-10, CIFAR-100, and ImageNet.Figure 1 distinguishes LWTA, where the winning unit can determine the outgoing weight, from maxout, which shares outgoing weights within a block.Source: §1 · p. 2, Figure 1 · p. 2

2 LOCALLY COMPETITIVE NEURAL NETWORKS

ReLU, LWTA, and maxout implement local competition that gates activation and error flow through part of a network. The resulting subnetworks can be viewed as linear models trained on different examples.

2 LOCALLY COMPETITIVE NEURAL NETWORKS

  • ReLU, LWTA, and maxout implement local competition, with ReLU competing against zero and LWTA and maxout competing within unit groups.In LWTA the winning unit’s output is gated by an indicator; maxout outputs the largest input in a block and can be viewed as LWTA with shared outgoing weights.Source: §2 · p. 2
  • For each input, non-zero activations define a subnetwork through which activations and training errors propagate.With ReLUs, different examples can activate many different subsets of units.Source: §2 · p. 2
  • Training can be viewed as training many linear subnetworks, with different examples routed to different subnetworks and the appropriate one activated at test time.The paper describes this as a way the system generalizes to a given example.Source: §2 · p. 2

3 SUBNETWORK ANALYSIS

The section represents each input by its active-unit subnetwork and examines how training organizes these subnetworks. It finds class-related structure, rapid stabilization, and useful kNN classification from submasks.

3 SUBNETWORK ANALYSIS

  • Submasks encode each example’s active units as a bitstring, enabling clustering analysis and use as descriptors for retrieval.The section uses t-SNE to visualize submask relationships and evaluates them on MNIST.Source: §3 · pp. 2–3

3.1 VISUALIZATION THROUGH DIMENSIONALITY REDUCTION

  • Trained ReLU submasks cluster by MNIST class, unlike those from a randomly initialized network; misplaced examples can fall in the wrong class cluster.Similar-class subnetworks are more alike than different-class subnetworks, and such misassignments indicate the wrong subnetwork was selected.Source: §3.1 · pp. 3–4, Figure 3 · p. 3
  • LWTA and maxout submasks show similar clustering; maxout masks mark the units with maximum activations, while the LWTA evidence is qualitative.For maxout, units are pooled before propagation, so the authors construct masks from the winning units.Source: §3.1 · p. 4

3.2 BEHAVIOR DURING TRAINING

  • Flip fractions start at 0.2 and quickly fall below 0.05, with about three epochs of reassignment before subnetworks are fine-tuned.The reported curve averages the fraction of patterns whose unit activation changes between epochs.Source: §3.2 · p. 4, Figure 4 · p. 4

3.3 EVALUATING SUBMASKS

  • MNIST kNN classification using submasks is close to softmax classification; using non-pooled maxout activations instead gives 121 errors.The table caption says the reported kNN results vary slightly across runs and were not cherry-picked.Source: Table 1 · p. 4, §3.3 · p. 5
  • On MNIST, convolutional maxout produces 52 errors with the network and 65 errors with kNN on its submasks.The submasks come from presynaptic activations in the penultimate convolutional layer.Source: §3.3 · p. 5

3.4 EFFECT OF DROPOUT

  • When dropout training continues beyond a validation error comparable to the no-dropout case, submask kNN performance improves as validation error falls.The authors interpret this as consistent with dropout making subnetwork organization more robust through injected noise.Source: §3.4 · p. 5

4 EXPERIMENTAL RESULTS

Experiments evaluate submasks on CIFAR-10, CIFAR-100, and ImageNet, finding useful classification and retrieval information in these binary representations. On ImageNet, they support compact storage and competitive retrieval, though supervised hashing ranks results better at high recall.

4.1 CIFAR-10 & CIFAR-100

  • On CIFAR-10 and CIFAR-100 test sets, nearest-neighbor classification using submasks loses, on average, 1.25% and 2.26% accuracy, respectively, compared with using unit activation values.The CIFAR experiments used maxout models and evaluated test-set classification with the softmax output layer and kNN on penultimate-layer unit activations and submasks.Source: §4.1 · p. 5, §4 · p. 5
  • CIFAR submask clusters are less separated than MNIST’s, and the poorer separation on CIFAR-100 corresponds to high classification error.The 10-cluster structure is visible for CIFAR-10; several smaller clusters remain visible for CIFAR-100.Source: Figure 5 · p. 6, §4.1 · p. 6

4.2 IMAGENET

  • The ImageNet evaluation uses submasks from a large convolutional network and compares them with supervised DiffHash trained on non-binarized network features.DiffHash supervision uses pairs labeled similar or dissimilar by whether they share a class.Source: §4.2 · p. 6
  • ImageNet submasks retain substantial information while improving storage efficiency; 4096-dimensional codes for the training set fit in about 0.5 GB.The codes are used directly for high-level-similarity retrieval without pair-wise training supervision.Source: §4.2 · p. 6
  • About 6% higher network accuracy accompanied about 10% higher submask accuracy on ImageNet.Table 3 compares two networks on validation-set top-5 classification accuracy.Source: Table 3 · p. 7
  • Up to mAP@100, ImageNet submasks show about a 3-point performance degradation; their retrieval curves decay against supervised hashing only at high recall.The caption attributes the high-recall ranking advantage to pair-wise supervision.Source: Table 4 · p. 7, Figure 6 · p. 7

5 DISCUSSION

The discussion interprets locally competitive networks as collections of subnetworks that self-organize during training. It connects this organization to reduced interference and compact binary descriptors derived from classification networks.

5 DISCUSSION

  • Local competition and parameter sharing let one network behave like many subnetworks without a global gating mechanism or modified competition objective.The paper contrasts this with training many specialized networks, which it describes as difficult without global gating and objective changes.Source: §5 · p. 7
  • During an early training phase, subnetworks for similar examples come to share more parameters, reducing interference from dissimilar examples; later tuning adjusts parameters with less reassignment.The authors describe this organization as enabling better training of subnetworks.Source: §5 · p. 7
  • The authors treat reduced interference as indirect evidence of a training benefit and relate subnetwork organization to the data-manifold hypothesis.They expect the organization of subnetworks responding to data points to reflect the modeled data manifold.Source: §5 · pp. 7–8
  • The paper presents submasks as binary descriptors obtained directly from classification networks, without additional training, and more storage- and retrieval-efficient than continuous activations.Figure 7 illustrates retrieval with a query image followed by five responses retrieved using submasks.Source: §5 · p. 8, Figure 7 · p. 8

A.1 EXTRA VISUALIZATIONS

The appendix provides additional two-dimensional views of submasks from trained and untrained networks on MNIST. The trained-network visualization shows class-specific clusters.

A.1 EXTRA VISUALIZATIONS

  • Trained three-hidden-layer LWTA and maxout networks show submasks organized into distinct, class-specific clusters on MNIST.The caption describes this organization as similar to that observed for ReLU networks.Source: Figure 8 · p. 10
  • The untrained-network visualization plots first-hidden-layer submasks from LWTA and ReLU networks on the MNIST test set.The caption identifies this layer as the one closest to the input.Source: Figure 9 · p. 10

B.1 CIFAR-10 AND CIFAR-100

CIFAR-10 and CIFAR-100 provide color-image evaluation datasets with different class counts, and the cited models use preprocessing and data augmentation.

B.1 CIFAR-10 AND CIFAR-100

  • CIFAR-10 contains 32×32 color images from 10 classes, with 50,000 training and 10,000 testing images.There are 6,000 images per class.Source: §B.1 · p. 10
  • CIFAR-100 is a similar color-image dataset with 100 classes and 600 images per class, making it more challenging.Source: §B.1 · p. 10
  • The cited models use global contrast normalization, ZCA whitening, and translational and horizontal-reflection data augmentation.Source: §B.1 · p. 10

B.2 IMAGENET (ILSVRC-2012)

The ImageNet experiments use penultimate-layer activations from an AlexNet implementation and retrieve validation examples by submask similarity.

B.2 IMAGENET (ILSVRC-2012)

  • ILSVRC-2012 contains over a million natural images across 1,000 classes.Source: §B.2 · p. 10
  • The experiments use downloaded penultimate-layer activations from a Krizhevsky et al. network implementation, with center-only 224×224 image crops.The implementation has some minor differences from the original.Source: §B.2 · p. 10
  • For each validation example, the 100 training examples with closest submasks are inverse-distance weighted to predict classes by top-1 or top-5 weighted sums.Source: §B.2 · p. 11

C NOTE ON SIGMOIDAL NETWORKS

The paper focuses on understanding locally competitive networks, while treating retrieval codes as a secondary outcome. It contrasts their inherent binarization with thresholded sigmoid activations, whose threshold requires tuning and whose networks are slow to train.

C NOTE ON SIGMOIDAL NETWORKS

  • The paper's primary focus is understanding locally competitive networks; obtaining binary retrieval codes from classification-trained networks is not its primary aim.Source: C NOTE ON SIGMOIDAL NETWORKS · p. 11
  • For sigmoid networks, activation thresholding is somewhat arbitrary and requires trying different values, whereas binarization in locally competitive networks is natural and inherent to credit assignment.Source: C NOTE ON SIGMOIDAL NETWORKS · p. 11
  • The paper notes that sigmoid networks are hard and slow to train, making activation thresholding impractical for large datasets common in retrieval tasks.Source: C NOTE ON SIGMOIDAL NETWORKS · p. 11
Loading 1410.1165v3…