PaperlayerView exact revision on arXiv

Source-linked AI summary

Generalized Shape Metrics on Neural Representations

Alex H. Williams, Erin Kunz, Simon Kornblith, Scott W. Linderman

arXiv:2110.14739v2stat.MLcs.LG

TL;DR

The paper addresses the lack of a unified framework for comparing neural representations across large collections of biological and artificial networks. It develops proper generalized shape metrics, convolution-aware measures, and Euclidean embeddings, then applies them to large neuroscience and deep-learning datasets. The resulting framework supports scalable analyses of representation geometry while retaining an important limitation: representational geometry may be only loosely tied to higher-level algorithmic principles of network function.

  • Problem

    Existing representation-similarity methods do not provide a unifying framework for choosing measures, using their outputs in downstream tasks, or extending them to large-scale analyses.

  • Method

    The paper defines generalized shape metrics, modifies CCA-based measures to satisfy the triangle inequality, introduces convolutional-layer metrics, and constructs approximate Euclidean embeddings.

  • Results

    The framework provides theoretical grounding for existing similarity measures and supports scalable machine-learning workflows demonstrated on Allen Brain Observatory and NAS-Bench-101 representations.

  • Takeaways & Limitations

    Proper metric spaces and Euclidean embeddings enable neural representations to be incorporated into a broad range of supervised and unsupervised analyses.

  • Takeaways & Limitations

    Representational geometry may be only loosely tied to higher-level algorithmic principles of network function.

Abstract

from arXiv · show

Understanding the operation of biological and artificial networks remains a difficult and important challenge. To identify general principles, researchers are increasingly interested in surveying large collections of networks that are trained on, or biologically adapted to, similar tasks. A standardized set of analysis tools is now needed to identify how network-level covariates -- such as architecture, anatomical brain region, and model organism -- impact neural representations (hidden layer activations). Here, we provide a rigorous foundation for these analyses by defining a broad family of metric spaces that quantify representational dissimilarity. Using this framework we modify existing representational similarity measures based on canonical correlation analysis to satisfy the triangle inequality, formulate a novel metric that respects the inductive biases in convolutional layers, and identify approximate Euclidean embeddings that enable network representations to be incorporated into essentially any off-the-shelf machine learning method. We demonstrate these methods on large-scale datasets from biology (Allen Institute Brain Observatory) and deep learning (NAS-Bench-101). In doing so, we identify relationships between neural representations that are interpretable in terms of anatomical features and model performance.

1 Introduction

The paper develops a metric-space framework for comparing neural representations at the scale of thousands of biological and artificial networks. It modifies existing measures, introduces convolution-aware metrics, and enables machine-learning analyses of representation geometry.

  • Existing representation-comparison methods have yielded insights but were not extended to systematic analyses spanning thousands of networks.
  • The framework develops dissimilarity measures that are proper metrics, satisfying symmetry and the triangle inequality.
  • Existing measures can often be modified to satisfy the triangle inequality, while novel metrics are specialized to convolutional layers.
  • Low-distortion Euclidean embeddings allow neural representations to serve as inputs to supervised and unsupervised machine-learning methods.
  • The methods are demonstrated on mouse visual-cortex representations and a database of deep artificial networks.
  • The work provides theoretical grounding for existing similarity measures and a general-purpose workflow that scales to thousands of networks.

2 Methods

The paper develops metric-space methods for comparing neural representations despite arbitrary neuron correspondences and varying network sizes. It generalizes shape-analysis ideas to define invariant distances, including a convolutional metric that preserves spatial structure.

  • Problem setup: Neural representations require comparison methods that account for arbitrary neuron ordering and differing numbers of neurons.Direct Frobenius distances fail when one representation is a permutation of the other, while existing methods lack a unified framework for downstream use.
  • Feature space mapping: Representations are mapped into a common p-dimensional feature space before defining distances between networks of unequal size.When network sizes differ, PCA is one example of a mapping into a common dimension.
  • Generalized shape metrics: Generalized shape metrics minimize distance over allowable transformations, using equivalence relations to encode symmetries such as neuron permutations, rotations, or linear alignments.The framework treats representations related by transformations in a group as equivalent and defines distances on the resulting quotient space.
  • Generalized shape metrics: Modifying CCA-based distances with arccos produces a proper metric, while regularization is typically needed when covariance matrices are ill-conditioned or singular.The transformed average canonical correlation satisfies the triangle inequality under the framework's metric construction.
  • Convolutional metrics: The convolutional metric permits orthogonal channel transformations but only spatial shifts, distinguishing spatial activation patterns while treating coherent translations as equivalent.Its equivalence relation uses circular shifts across height and width together with an orthogonal transformation across channels.

3 Applications and Results

The proposed shape metrics correct triangle-inequality violations in neural-representation comparisons and support low-distortion Euclidean embeddings for large-scale analyses. Applications to brain regions and neural networks show anatomically interpretable structure, cross-layer persistence, and predictive information about network accuracy.

  • Metric validation: 17/1128 ABO network pairs and 10128/100000 NAS-Bench-101 Stem-layer pairs exhibited triangle-inequality violations under the linear heuristic.Standard RSA also showed violations in 14/1128 ABO network pairs.
  • Metric validation: ~85.1% dendrogram similarity resulted when hierarchical clustering used the linear heuristic rather than the shape distance.Violation rates were approximately 1% in ABO and 10% in the NAS-Bench-101 Stem layer.
  • Euclidean embeddings: 95% of pairwise distances were distorted by at most ~5% for ABO or 10% for NAS-Bench-101 at sufficiently large embedding dimensions.Moderate dimensions around L ≈20 produced high-quality Euclidean embeddings, whereas L = 2 caused highly distorted geometry.
  • ABO applications: PCA of 48 ABO brain regions separated cortical and sub-cortical regions and clustered different layers of the same region.Embedded representations also supported prediction of anatomical hierarchy scores.
  • NAS-Bench-101 applications: Across 2000 NAS-Bench-101 networks, global representational structure was preserved across five layers, and early-layer embeddings moderately predicted test-set accuracy.The result included the Stem layer, whose architecture is identical across networks but receives different backpropagated gradients.

4 Conclusion and Limitations

The paper grounds neural-representation analyses in proper metric spaces and demonstrates scalable modeling approaches. It cautions that representational geometry may be only loosely tied to higher-level algorithmic principles and that current metrics omit topological features.

  • Contributions: Proper metric spaces provide theoretical advantages for analyzing neural representations, while Euclidean embeddings offer practical modeling approaches.The authors demonstrate scalability by analyzing thousands of networks.
  • Limitations: Representational geometry may be only loosely tied to higher-level algorithmic principles of network function.The authors suggest it may instead provide insight into lower-level implementational principles.
  • Limitations: The analyses use extensions of regularized CCA, while nonlinear metrics remain under-investigated and may capture additional structures.The authors note that currently popular linear methods may be insufficient for some structures of interest.
  • Limitations: Generalized shape metrics do not directly capture topological features of neural representations.The authors identify topological data analysis as a possible direction for developing such metrics.
  • Limitations: Several described metrics can be viewed as geodesic distances on Riemannian manifolds, but optimized Euclidean embeddings provide a practical approximate solution for large-scale surveys.The authors note that methods rigorously adapted to these manifolds would be desirable.

Supplemental Information: Generalized Shape Metrics on Neural Representations

The supplement organizes mathematical background, proofs, generalized metric results, and extensions for stochastic neural responses. It also establishes notation for vectors, matrices, linear operators, and abstract vector spaces.

  • Organization: The supplement is organized into sections covering metric-space background, main-text propositions, generalized shape metrics, and extensions.It includes results involving CCA, kernel CCA, geodesic distance on Kendall’s shape space, and stochastic layers.
  • Notation: Vectors use boldface lowercase letters, matrices use boldface uppercase letters, and linear operators follow the same uppercase notation.The notation distinguishes these objects from regular-type scalars or abstract vector-space elements.
  • Notation: Regular-type symbols may denote scalars or abstract vector-space elements, including random variables over R^n.The framework extends shape metrics to stochastic layers and neural responses.
  • Notation: Linear-operator notation uses juxtaposition for application and composition, paralleling matrix-vector and matrix-matrix multiplication.For example, T_1T_2X denotes T_1(T_2(X)).

A.2 Metrics

This section defines metric-space foundations and explains how Euclidean and angular distances behave in Hilbert spaces. It also introduces equivalence classes, isometries, and orthogonal transformations relevant to generalized shape metrics.

  • Metric spaces: A metric satisfies identity, symmetry, nonnegativity, and the triangle inequality; relaxing identity to an equivalence relation yields a metric on a quotient space.The quotient construction groups elements considered equivalent under the specified relation.
  • Angular distance: Angular distance is not a metric on all of R^n but becomes a metric over equivalence classes defined by nonnegative scaling.Geometrically, it is the geodesic path length between points on a sphere.
  • Hilbert spaces: Hilbert spaces generalize Euclidean vector spaces and matrix spaces, with inner products supporting norms, Cauchy-Schwarz, and sub-additivity.Random vectors over R^n also form a Hilbert space under an expected dot-product inner product.
  • Euclidean distance: Euclidean distance satisfies the triangle inequality by applying norm sub-additivity to X − M + M − Y.It also satisfies symmetry and nonnegativity.
  • Angular distance: The angular-distance proof decomposes X and Z into components parallel and orthogonal to Y, then uses Cauchy-Schwarz to establish θ_XZ ≤ θ_XY + θ_YZ.Monotonicity of arccos converts the cosine bound into the angular triangle inequality.
  • Isometries: Orthogonal transformations preserve inner products and therefore act as isometries for generalized Euclidean and angular distances.They are closed under inverses and composition, forming a group.

B Proof of Propositions 1 & 2

The proof establishes that minimizing a metric over a group of isometries produces a metric on the corresponding quotient space. This general result supports the paper’s specialized propositions and alignment-based distances.

  • General proposition: A general proposition states that minimizing a metric over a group of isometries defines a metric on the quotient space induced by the group action.The equivalence relation is X ∼ Y exactly when X = T Y for some T in the group.
  • Proof: The proof establishes identity through the optimal alignment, showing h(X,Y)=0 exactly when X and Y are related by a group transformation.This uses the identity property of the underlying metric.
  • Proof: Symmetry follows from invertibility of group elements and the isometry property, allowing optimal alignments in either direction to be compared.The two inequalities imply h(X,Y)=h(Y,X).
  • Proof: The triangle-inequality proof replaces an optimal alignment with a composed alignment, applies the triangle inequality of g, and uses isometry invariance.The midpoint is chosen through the aligned representation of the intermediate element.
  • Special cases: For Proposition 1, the underlying space is R^{m×p} with Euclidean distance and a group of right-multiplying linear isometries.For Proposition 2, the space is the unit-Frobenius sphere with angular distance and the same group structure.

C Connections to Other Methods

The framework connects neural-representation comparisons to shape analysis by treating alignment under permutations or orthogonal transformations as metric constructions. These formulations reduce to standard optimization problems with efficient or closed-form solutions.

  • Representations are compared as activation matrices over common inputs, with PCA or zero-padding accommodating networks having different numbers of units.Mean-centering can remove translations in activation space before comparison.
  • Permutation alignment minimizes Euclidean distance between activation matrices and can be reformulated as a linear assignment problem.The assignment problem is solvable in O(n^3) time using standard algorithms.
  • Orthogonal alignment solves the Procrustes problem by optimizing over orthogonal transformations between representation matrices.The solution is obtained from the singular value decomposition of XᵀY and maximizes the corresponding inner product.
  • The Procrustes optimum is characterized by the singular values of XᵀY, whose sum equals the optimal objective value.The proof uses the orthogonality of the transformed matrix and the nonnegativity of the singular values.

C.3 Canonical Correlation Analysis (CCA)

CCA is recast as an orthogonal Procrustes problem after whitening the two representation matrices, placing canonical correlations within the generalized metric framework. Regularized CCA then interpolates continuously between CCA and Procrustes alignment, while kernelization extends the approach to nonlinear feature spaces.

  • Canonical Correlation Analysis (CCA): CCA chooses transformations that maximize correlation between XW_x and YW_y subject to normalization constraints.The constraints prevent arbitrary rescaling from making the objective unbounded.
  • Canonical Correlation Analysis (CCA): Whitening transforms CCA into a Procrustes problem on X_φ and Y_φ with orthogonal constraints.The transformed matrices have identity Gram matrices, and the canonical correlations are the singular values of X_φᵀY_φ.
  • Canonical Correlation Analysis (CCA): The resulting CCA dissimilarity is a metric because normalized representations have unit Frobenius norm and orthogonal transformations act as isometries.This connects the CCA construction directly to the metric-space guarantees of the framework.
  • Regularized CCA: Regularized CCA uses α to interpolate between unregularized CCA at α = 0 and fully regularized Procrustes alignment at α = 1.A partial whitening transformation reduces the regularized optimization to Procrustes alignment.
  • Kernel CCA: Kernel CCA generalizes CCA through feature mappings into high-dimensional or infinite-dimensional reproducing kernel Hilbert spaces.With a linear kernel, the kernel formulation is equivalent to classic CCA up to a change of variables.
  • Shape-space connections: The framework also considers special-orthogonal alignment and connects its resulting distances to Riemannian distances in size-and-shape and Kendall’s shape spaces.These connections arise when the transformation group is the special orthogonal group SO.

C.7 Centered Kernel Alignment (CKA) and Representational Similarity Analysis (RSA)

CKA and RSA compare transformed representations through matrices such as XXᵀ and YYᵀ, but their original similarity formulations do not generally satisfy metric-space requirements. Applying angular distance, including arccos, converts CKA into a proper metric while preserving its covariance-based comparison structure.

  • Representational comparisons: Linear CKA and RSA compare XXᵀ with YYᵀ, which represent covariance structure across common test inputs after mean-centering.RSA may instead use pairwise Euclidean distances between activation patterns to form representational distance matrices.
  • Metric limitations: Spearman-correlation RSA does not produce a metric, and originally formulated CKA also lacks the metric-space guarantee.The issue is that these similarity measures need not satisfy the triangle inequality.
  • Metric reformulation: Applying arccos to CKA yields a proper metric based on angular distance over the associated matrix representations.The construction uses Frobenius norms and can be instantiated for linear CKA.
  • Comparison with CCA: The CKA-based metric differs from fully regularized CCA by using a squared Frobenius norm and covariance-matrix normalization rather than a nuclear norm and representation-matrix normalization.These choices produce related but distinct measures of representational alignment.
  • Alternative metric construction: An alternative approach modifies CKA and RSA using the Riemannian metric over positive-definite matrices.The generalized angular-distance construction instead applies to matrices that need not be positive-definite.

D Probabilistic interpretations of generalized shape metrics

Generalized shape metrics admit a probabilistic interpretation by treating neural responses as random vectors and defining their inner products through expectations. Finite input samples yield computable approximations, while continuous input distributions expose a sampling-based limitation and motivate extensions to stochastic representations.

  • Random-vector formulation: Random neural representations can be treated as elements of a vector space with an expectation-based inner product.The construction establishes the Hilbert-space foundation needed to define metrics over random vectors.
  • Discrete inputs: For deterministic responses sampled uniformly from m inputs, the probabilistic inner products coincide with scaled Frobenius inner products of the response matrices.Thus, the finite-sample generalized shape metrics can be interpreted as distances over uniformly sampled inputs.
  • Continuous inputs: For continuous input distributions, the relevant inner product is an integral over input patterns that is generally intractable because the density may be unknown.Independent samples can approximate the integral through a finite-sample average.
  • Approximation quality: When test-set statistics match the real-world input distribution, the sample-based estimate can appropriately approximate the true representational distance.The sample size m controls the approximation studied in Figures 3B and 3C.
  • Stochastic representations: The framework can be extended to stochastic neural representations, including biological noise and dropout, but this extension is presented as future research.The probabilistic framing is what enables this proposed direction.

E.1 Experiments on sample size (Fig. 3)

The experiments compare representation analyses across convolutional layers and biological neural recordings, using alignment, interpolation, clustering, and regression procedures. They also examine how 2D versus higher-dimensional MDS changes visualization of the ABO dataset.

  • Convolutional networks: Activations were sampled from three layers following stride-2 convolutions, with smaller feature maps upsampled by linear interpolation before comparison.A brute-force search covered circular shifts along width and height dimensions.
  • Convolutional networks: The remaining Figure 3 analyses used activations from the final layer before average pooling.
  • Biological recordings: Allen Brain Observatory units were pooled across sessions, anatomically assigned using the reference atlas, and converted into spike counts.
  • Biological recordings: Hierarchical clusterings were computed with Ward’s linkage criterion and visualized using SciPy tools.
  • Biological recordings: Kernel ridge regression predicted anatomical hierarchy scores across 29 brain regions, excluding PO and VPM as outliers.

E.3 NAS-Bench-101

The NAS-Bench-101 analysis evaluates representations from many trained architectures and relates their geometry to downstream modeling choices. It uses a fixed cell-based architecture space and compares low-dimensional embedding procedures for the ABO dataset.

  • Dataset and computation: Representations were computed for 2,000 randomly selected NAS-Bench-101 architectures trained for 108 epochs, requiring 260 core-years of computation.
  • Regression analysis: Ridge-regression analyses used 80% of data for training, 10% for validation, and 10% for computing R2.The validation set selected the ridge hyperparameter and kernel bandwidth.
  • Embedding comparison: MDS directly to L = 2 yields a distinct visualization from MDS to L = 20 followed by PCA projection to 2D.The L = 20 procedure is advocated for downstream modeling because it better approximates the true metric space.
  • Architecture: The NAS-Bench-101 architecture contains repeated cells selected from a fixed space, with channel counts differing by stack.Figure 5 analyzes representations from the outputs of the highlighted layers.
Loading 2110.14739v2…