PaperlayerView exact revision on arXiv

Source-linked AI summary

Analyzing and Improving Representations with the Soft Nearest Neighbor Loss

Nicholas Frosst, Nicolas Papernot, Geoffrey Hinton

arXiv:1902.01889v1stat.MLcs.LG

TL;DR

The paper asks how class-manifold similarity in learned representations can be measured and improved. It expands the soft nearest neighbor loss and studies its use for analyzing and training representations. Maximizing hidden-layer entanglement improves generalization and supports better uncertainty estimates for outlier data, which can be identified by unusually few predicted-class neighbors.

  • Problem

    Representation similarity affects generalization and confidence, but robustly analyzing class-manifold entanglement and outlier support remains important for models facing unfamiliar inputs.

  • Method

    The paper expands the soft nearest neighbor loss with temperature and uses it to analyze representations and as a training-objective bonus that maximizes hidden-layer entanglement.

  • Results

    Maximizing hidden-layer entanglement improves generalization and produces representations that better support uncertainty estimates on adversarial and out-of-distribution data.

  • Takeaways & Limitations

    Outlier data can be recognized because hidden layers contain fewer than the normal number of neighbors from the predicted class.

  • Takeaways & Limitations

    The reported baseline accuracies are below current state of the art because the experiments intentionally isolate the soft nearest neighbor loss from other regularizers.

Abstract

from arXiv · show

We explore and expand the $\textit{Soft Nearest Neighbor Loss}$ to measure the $\textit{entanglement}$ of class manifolds in representation space: i.e., how close pairs of points from the same class are relative to pairs of points from different classes. We demonstrate several use cases of the loss. As an analytical tool, it provides insights into the evolution of class similarity structures during learning. Surprisingly, we find that $\textit{maximizing}$ the entanglement of representations of different classes in the hidden layers is beneficial for discrimination in the final layer, possibly because it encourages representations to identify class-independent similarity structures. Maximizing the soft nearest neighbor loss in the hidden layers leads not only to improved generalization but also to better-calibrated estimates of uncertainty on outlier data. Data that is not from the training distribution can be recognized by observing that in the hidden layers, it has fewer than the normal number of neighbors from the predicted class.

1. Introduction

The paper expands the soft nearest neighbor loss to analyze class-manifold entanglement and finds that maximizing entanglement in hidden feature-extraction layers can improve generalization, discrimination, and outlier uncertainty estimates.

  • Motivation: The paper motivates representation structure as important for classifier generalization and calibrated confidence when outlier data may occur.The examples include medical diagnosis and safety- or security-sensitive settings.
  • Contribution: The soft nearest neighbor loss measures the entanglement of class manifolds and supports analysis of representation learning.The paper applies it to discriminative and generative training.
  • Entangled hidden representations: Maximizing entanglement in hidden feature-extraction layers benefits final-layer discrimination, unlike the penultimate layer.The authors add the loss as a bonus to the training objective rather than forcing hidden representations to separate classes.
  • Entangled hidden representations: The entanglement bonus regularizes models by making already-similar representations more similar when their labels differ.These representations form class-independent clusters that capture similarity useful for later discrimination.
  • Outlier uncertainty: Entangled representations support better uncertainty estimates for adversarial examples and test inputs from different distributions.Out-of-distribution data has fewer than the normal number of predicted-class neighbors in hidden layers.

2. Soft Nearest Neighbor Loss

The soft nearest neighbor loss quantifies how likely a point is to select a same-class neighbor, with temperature controlling distance sensitivity. Its behavior differs from triplet loss when maximized, while minimizing it can preserve multimodal class structure.

  • Interpretation: Low entanglement means representations are closer to same-class points than different-class points, supporting accurate nearest-neighbor classification.The loss therefore compares within-class and between-class proximity in representation space.
  • Loss definition: The soft nearest neighbor loss is the negative log probability of sampling a same-class neighbor, with probabilities determined by pairwise distances.It uses all positive and negative points in the batch and introduces temperature to control distance weighting.
  • Loss definition: The input x may be raw data or a hidden-layer representation, allowing the loss to analyze similarity structures throughout a model.At low temperatures, small distances dominate the loss.
  • Multimodal structure: Minimizing entanglement does not necessarily collapse each class to one point; several widely separated pure clusters can remain.A second mode in each class is preserved under gradient descent.
  • Comparison with triplet loss: Unlike triplet loss, which samples one positive and one negative point, the soft nearest neighbor loss uses all positive and negative batch points.When maximized, this produces representations more spread out than those produced by triplet loss.
  • Temperature: Temperature divides squared distances before negative exponentiation, so larger temperatures allow widely separated points to influence the loss.The paper later removes temperature as a hyperparameter by minimizing the loss over temperatures and approximates this with gradient descent.
  • Comparison with triplet loss: Minimizing either the triplet or soft nearest neighbor loss separates four classes by a larger margin, whereas maximizing them yields different geometries.Triplet loss densely projects points around a circle, while soft nearest neighbor loss spreads them throughout representation space.

3. Measuring Entanglement during Learning

The soft nearest neighbor loss tracks how class and data manifolds become entangled during learning. In classifiers, lower hidden layers become more entangled while the final layer remains discriminative; in generative models, real and synthetic data increasingly overlap, though pixel-space optimization does not generalize to CIFAR10.

  • Analytical use: The soft nearest neighbor loss characterizes class similarity structures throughout discriminative and generative learning.It measures classifier-layer entanglement and real–synthetic data entanglement during generative training.
  • Discriminative Models: Lower ResNet layers initially disentangle CIFAR-10 data, then become increasingly entangled as features represent subfeatures shared across classes.This trend occurs throughout the block except at its output layer.
  • Discriminative Models: The final hidden layer becomes progressively less entangled because it must support linear separation of the correct-class logit from the others.The block output remains discriminative while lower layers develop shared features.
  • Generative Models: During vanilla GAN training on CIFAR10, real and synthetic data become less separable, with increasing t-SNE overlap and soft nearest neighbor loss.Some input modes remain ignored by the generator, while some generated modes do not represent the true distribution.
  • Generative Models: The loss can replace the GAN discriminator as a generator objective, producing realistic and varied MNIST images but noticeably smoother samples.The approach captures all classes and multiple class modes, including curly and straight “2”s.
  • Generative Models: Soft nearest neighbor optimization does not generalize to CIFAR10 when entanglement is measured with Euclidean distance in pixel space.The authors suggest measuring entanglement in a learned space instead; a proof-of-concept succeeds on MNIST.

4. Entangling Representation Spaces

The paper argues that maximizing entanglement in hidden representations, rather than forcing class separation, regularizes learning and can improve generalization. A composite objective combining cross-entropy with the soft nearest neighbor loss reduces overfitting and marginally improves test performance.

  • 4.1. Intuition behind Maximizing Entanglement: Maximizing entanglement challenges the assumption that hidden representations should separate classes, instead encouraging class-independent similarity structures.The authors distinguish feature-extracting hidden layers from the penultimate layer, where separation remains beneficial.
  • 4.1. Intuition behind Maximizing Entanglement: The composite objective minimizes cross-entropy while maximizing the soft nearest neighbor loss on hidden representations through a negative entanglement weight α.The network is represented as transformations f_k, with the soft nearest neighbor term applied to hidden layers and cross-entropy applied to logits.
  • 4.2. Soft Nearest Neighbor Loss as a Regularizer: All entangled models outperformed their non-entangled counterparts to some extent, marginally increasing test performance across the studied datasets.The experiments used CNNs for MNIST, Fashion-MNIST, and SVHN, and a ResNet for CIFAR10, without additional regularizers.
  • 4.2. Soft Nearest Neighbor Loss as a Regularizer: The reported baselines were intentionally below state-of-the-art so the soft nearest neighbor loss could be studied without other regularizers.This design isolates the loss from factors such as dropout that could also affect representation spaces.
  • 4.2. Soft Nearest Neighbor Loss as a Regularizer: The CIFAR10 hyperparameter search independently confirmed that maximizing entanglement led to better test performance.The search varied learning rate, entanglement hyperparameter α, and initial temperature T across 64 ResNet v2 training runs.
  • 4.2. Soft Nearest Neighbor Loss as a Regularizer: Entangled models reduced overfitting, with the SVHN model beginning to overfit about 5,000 training steps later and to a much lesser degree.The learning curves compare cross-entropy-only and entangled variants trained for many steps.

5. Entangled Models in Adversarial Settings

The paper evaluates entangled representations for uncertainty estimation and adversarial settings using Deep k-Nearest Neighbors and several out-of-distribution tests. Entangled models provide more accurate credibility–accuracy relationships and better separate outliers, while reducing gradient alignment across models.

  • 5.1. Entangled Representations support more Calibrated DkNN Estimates of Uncertainty: DkNN estimates uncertainty by searching each hidden-layer representation space for nearby training points and comparing their labels with the predicted label.Agreement among nearest training labels yields high credibility; disagreement yields low credibility.
  • 5.1. Entangled Representations support more Calibrated DkNN Estimates of Uncertainty: Entangled models consistently produced better-calibrated DkNN credibility estimates than standard models across white-box and black-box adversarial settings.The correlation between credibility and classification accuracy was consistently higher for entangled models, although it was not perfectly linear.
  • 5. Entangled Models in Adversarial Settings: The adversarial evaluation covered white-box FGSM and BIM attacks, plus black-box BIM examples transferred from another model.The setup tests both direct gradient access and transfer-based attacks, including gradient-masking concerns.
  • 5. Entangled Models in Adversarial Settings: Entangled models better separated adversarial data from real data in activation space, allowing outliers to be recognized by their unusually few predicted-class neighbors.The authors explicitly distinguish this uncertainty benefit from robustness to adversarial examples.
  • 5. Entangled Models in Adversarial Settings: Entangled models disrupted gradient alignment across model pairs, making universal or transferable adversarial perturbations harder to find.The effect was observed when comparing standard with entangled models and when comparing two entangled models.
  • 5. Entangled Models in Adversarial Settings: On a different test distribution, the authors tested networks trained on SVHN using CIFAR10 inputs to examine out-of-distribution representation behavior.The experiment compares cross-entropy-only and entanglement-trained models through t-SNE visualizations of their logits.

6. Conclusions

The soft nearest neighbor loss supports improved generalization and uncertainty estimates by encouraging entangled representations that separate outlier data more consistently.

  • The loss can boost test performance and generalization when added as a bonus to a classifier’s training objective.
  • Entangled representations separate outlier data more consistently from real data in activation space.
  • Outlier data can be rejected by observing that it has fewer neighbors from the predicted class.
  • FGSM gradient visualizations compare two vanilla models with two entangled models, using distinct colors for each model pair.
  • Out-of-distribution data is easier to separate from in-distribution data for entangled models than for baseline models.

A. Soft Nearest Neighbor Loss on Toy distribution

On a toy distribution with two modes per class, minimizing entanglement preserves both initial modes rather than collapsing each class to one point.

  • Minimizing entanglement preserves the two modes initially present in each class.
  • Gradient descent on point coordinates reduces entanglement without collapsing each class into a single point.

B. Comparing the Soft Nearest Neighbor Loss with the Triplet Loss

The triplet loss and soft nearest neighbor loss similarly constrain relative within-class and between-class distances when minimized, but produce different representations when maximized.

  • Both losses measure relative distances between same-class and different-class points.
  • Triplet loss uses sampled positive and negative points, whereas soft nearest neighbor loss uses all points in a batch.
  • Minimizing either loss separates classes by a larger margin.
  • Maximizing the two losses produces different representations: triplet loss densely projects data around a circle, while soft nearest neighbor loss yields a different result.
  • Triplet-loss regularization did not improve DkNN calibration, unlike models maximizing entanglement with the soft nearest neighbor loss.

C. Additional Entanglement Measurements

Additional measurements show that soft nearest neighbor loss tracks representation changes during training, with dataset- and layer-dependent entanglement patterns and applications to generative models.

  • The loss can monitor entanglement during learning in models trained to minimize cross-entropy.
  • MNIST: On MNIST, the loss decreases throughout training, likely because the dataset is easier to separate in the input domain.
  • CIFAR10: On CIFAR-10, entanglement remains fairly constant or increases in the first three residual blocks during training.
  • CIFAR10: Entanglement remains high in lower residual blocks and decreases only in the final block.
  • Generative models: The loss measures entanglement between synthetic and real data across GAN variants.

E. Does Entanglement conflict with Robustness?

The experiments compare entanglement in models trained on original versus adversarial data and report DkNN credibility as a function of prediction accuracy.

  • The study reproduces PGD adversarial training and measures hidden-layer entanglement after minimizing cross-entropy on adversarial examples.
  • DkNN uncertainty calibration is reported for entangled MNIST and FashionMNIST models using credibility as a function of prediction accuracy.The plots follow the experiment described in Section 5.1.
  • The comparison measures entanglement loss at temperature T = 100 on batches of 128 MNIST test points for baseline and PGD models.

G. Out-of-Distribution Test Inputs

Entangled representations separate out-of-distribution notMNIST inputs from MNIST representations and separate adversarial from real test data, supporting more calibrated uncertainty estimates.

  • The MNIST and notMNIST experiment repeats an earlier SVHN and CIFAR10 analysis of separating test data from a different distribution.
  • Training with entanglement separates notMNIST inputs into a distinct cluster while preserving MNIST clusters, unlike the vanilla model.The vanilla model projects notMNIST inputs close together and makes confident MNIST-class predictions.
  • Entangled models separate real and adversarial test data more effectively than non-entangled cross-entropy models.In the non-entangled model, adversarial data is projected close to real test data.
  • Better separation improves the estimate of training neighbors supporting predictions, enabling more calibrated DkNN uncertainty estimates.

I. Soft Nearest Neighbor GANs

The paper applies the soft nearest neighbor loss to GANs and examines entanglement between synthetic and real data across architectures and learned representations.

  • The loss can replace the discriminator in an MNIST GAN, but the same setup did not scale to CIFAR10.The authors hypothesize that l2 distance characterizes MNIST inputs better than CIFAR10 inputs.
  • The study reports DkNN credibility as a function of prediction accuracy on MNIST and FashionMNIST models.
  • For cross-entropy-only models, adversarial examples generated on entangled models are transferred to assess DkNN credibility.The passage notes that entangled models make poor source models for transferability-based black-box attacks.
  • t-SNE visualizations compare in-distribution MNIST with out-of-distribution notMNIST representations in vanilla and entangled models.
  • On FashionMNIST, entangled models project adversarial data into a separate activation-space area, unlike vanilla models.In the vanilla model, adversarial and real data occupy a similar part of activation space.
  • An MNIST GAN experiment uses the soft nearest neighbor loss in a learned 10-dimensional discriminator space, with the discriminator minimizing and generator maximizing it.The experiment demonstrates use in learned space as well as pixel space.
Loading 1902.01889v1…