Insights on representational similarity in neural networks with canonical correlation

Ari S. Morcos; Maithra Raghu; Samy Bengio

Source-linked AI summary

Insights on representational similarity in neural networks with canonical correlation

Ari S. Morcos, Maithra Raghu, Samy Bengio

arXiv:1806.05759v3Jun 14, 2018stat.MLcs.AIcs.CVcs.LGcs.NE

TL;DR

Comparing neural-network representations across architectures and training time is difficult because their structures can differ substantially. The paper develops projection weighted CCA to separate signal from noise and compare representations across CNNs and RNNs. It finds systematic convergence patterns in CNNs and during RNN training, alongside substantial RNN variation across sequence timesteps.

Problem
Comparing neural-network representations across networks and over time is difficult because their structures vary substantially, even for networks trained on identical tasks.
Method
The paper develops projection weighted CCA, improving CCA by distinguishing stable signal from unstable noise in neural-network representations.
Results
Networks that generalize converge to more similar solutions than those that memorize; wider networks converge more similarly than narrow ones; RNNs converge bottom-up during training but vary across sequence timesteps.
Takeaways & Limitations
CCA provides a tool for comparing representations across different networks and across time, revealing structured similarities and diversity in CNNs and RNNs.
Takeaways & Limitations
Across sequence timesteps, even CCA often fails to find linear similarity in RNN hidden states, and the study’s RNN sequence-time investigation is preliminary.

Abstract

from arXiv · show

Comparing different neural network representations and determining how representations evolve over time remain challenging open questions in our understanding of the function of neural networks. Comparing representations in neural networks is fundamentally difficult as the structure of representations varies greatly, even across groups of networks trained on identical tasks, and over the course of training. Here, we develop projection weighted CCA (Canonical Correlation Analysis) as a tool for understanding neural networks, building off of SVCCA, a recently proposed method (Raghu et al., 2017). We first improve the core method, showing how to differentiate between signal and noise, and then apply this technique to compare across a group of CNNs, demonstrating that networks which generalize converge to more similar representations than networks which memorize, that wider networks converge to more similar solutions than narrow networks, and that trained networks with identical topology but different learning rates converge to distinct clusters with diverse representations. We also investigate the representational dynamics of RNNs, across both training and sequential timesteps, finding that RNNs converge in a bottom-up pattern over the course of training and that the hidden state is highly variable over the course of a sequence, even when accounting for linear transforms. Together, these results provide new insights into the function of CNNs and RNNs, and demonstrate the utility of using CCA to understand representations.

1 Introduction

Comparing population representations across neural networks is difficult because networks can implement apparently distinct solutions without one-to-one unit mappings. The paper uses CCA to compare such representations and identifies improvements and applications to CNNs and RNNs.

Population-level comparison is difficult because networks can converge to distinct solutions without one-to-one mappings between units.
CCA can reveal shared structure between superficially dissimilar representations because it is invariant to linear transforms.
The paper’s contributions include improving CCA’s separation of signal from noise and applying it to neural-network representations.
Networks that generalize converge to more similar solutions than networks that memorize, while wider networks converge more similarly than narrower networks.
RNN representations show bottom-up convergence during training but vary significantly across sequence timesteps.

2 Canonical Correlation Analysis on Neural Network Representations

CCA compares neural-network layer representations by finding correlated directions across activation vectors, while remaining invariant to invertible affine transforms. The paper extends mean CCA with projection weighting to distinguish task-relevant signal from noise and assess representational convergence.

Canonical Correlation Analysis: CCA relates two layers by finding maximally correlated, mutually orthogonal linear directions between their neuron activation vectors.Each activation vector contains one neuron’s outputs over a dataset, and a layer supplies a set of such vectors.
Canonical Correlation Analysis: CCA similarity is invariant to invertible affine transforms, enabling comparisons between superficially different network representations and across RNN training times.
Representational Dynamics: Task performance can converge before most CCA coefficients do, while early-stabilizing directions remain stable and late-changing directions become uncorrelated noise.The paper therefore treats early-stabilizing directions as signal and later-changing directions as unnecessary for task performance.
Beyond Mean CCA Similarity: Mean CCA can underestimate similarity because it treats all canonical directions as equally important even though networks may not use a layer’s full dimensionality.
Beyond Mean CCA Similarity: In a toy signal-noise comparison, projection-weighted CCA remains largely robust as signal-to-noise ratio decreases, unlike the unweighted mean.SVCCA performs better than the unweighted mean but worse than projection weighting.
Beyond Mean CCA Similarity: Projection weighting assigns greater weight to canonical correlations whose directions account for more of the original layer outputs, addressing the equal-importance assumption.The method explicitly orthonormalizes the CCA vectors before measuring how much original output each direction accounts for.

3 Using CCA to measure the similarity of converged solutions

The paper uses projection weighted CCA to compare converged CNN representations across training conditions, revealing systematic differences between generalizing, memorizing, wider, narrower, and differently trained networks.

Experimental setup: Projection weighted CCA compares representations across CNNs trained with different labels, widths, initializations, and learning rates.The representation for each layer was obtained by averaging across spatial locations within each filter.
Generalizing versus memorizing networks: Generalizing networks converge to more similar solutions than memorizing networks, although memorizing networks are as similar to one another as to a generalizing network.This pattern is especially pronounced in later layers; early layers converge to equally similar solutions across both groups.
Generalizing versus memorizing networks: Traditional cosine and Euclidean distances miss differences that CCA detects because networks can learn different linear transforms of similar solutions.Unweighted CCA captures the broad pattern but misses increased similarity among generalizing networks in the final two layers.
Network width: Larger networks converge to much more similar solutions than smaller networks.The comparison used groups of networks with increasing numbers of filters and controlled final-layer subsampling.
Network width: The correlation between test accuracy and pairwise CCA distance was -0.96 across network pairs.This suggests that CCA distance computed using training data may help predict test accuracy without a validation set.
Learning rates and solution clusters: Across 200 networks with identical topology and varying learning rates, CCA revealed five distinct solution clusters that aligned nearly perfectly with clusters found using ablation robustness.The networks had similar training loss, and many also reached similar test accuracy, while traditional generalization metrics obscured several differences.

4 CCA on Recurrent Neural Networks

The paper applies CCA to LSTMs to compare representations across both training time and sequence time. It finds bottom-up convergence during training, while hidden-state representations often vary nonlinearly across sequence timesteps.

Experimental setup: The RNN experiments use LSTMs trained on Penn Treebank and WikiText-2 language-modeling tasks.The analysis considers both training time, which changes weights, and sequence time, which changes cell and hidden states.
Sequence dynamics: Across sequence timesteps, CCA finds some similarity missed by traditional metrics, but often fails to find linear similarity in hidden states.The preliminary sequence-time analysis therefore indicates that hidden-state representations can vary nonlinearly over a sequence.
Training dynamics: CCA reveals bottom-up convergence during training on both Penn Treebank and WikiText-2.Early layers converge to their final representations before later layers, including in a deeper WikiText-2 LSTM.
Training dynamics: The weighted mean CCA reveals bottom-up convergence more clearly than the unweighted mean.Control comparisons with cosine and Euclidean distance also show bottom-up convergence, but less clearly than CCA.

5 Discussion and future work

The discussion frames projection weighted CCA as a tool for studying stable and unstable components of neural representations and summarizes distinct CNN solution patterns and RNN dynamics. It also identifies unresolved questions about preserved directions, cluster attributes, and temporal stability.

Discussion: Hidden-layer representations contain signal components stable over training and unstable noise components, motivating projection weighted CCA.The method adapts CCA by weighting components to distinguish these representational parts.
Discussion: CNNs that generalize, wider CNNs, and CNNs with different learning rates exhibit distinct patterns of representational convergence.The reported patterns include greater similarity for generalizing and wider networks, alongside diverse clusters across learning rates and initializations.
Discussion: RNNs show bottom-up convergence during training, while their representations vary nonlinearly across sequence timesteps.The discussion treats these as complementary temporal properties of recurrent representations.
Future work: Future work should identify what makes preserved directions special, what distinguishes the diverse CNN solution clusters, and which RNN representation aspects remain stable over time.The paper also proposes testing whether similarity can regularize performance and whether related findings extend from CNNs to RNNs.

A.1 Performance Plots for Models

Performance can reach near-final levels before most CCA coefficients converge, indicating that later-changing representation components may be unnecessary noise.

Performance is almost equivalent to final performance at a training time t0, while most CCA coefficients ρ(i) remain unconverged.The associated vectors are therefore suggested to be noise that is not necessary for doing well at the task.
Figure A1 reports performance convergence for CIFAR-10 CNNs and PTB and WikiText-2 RNNs.

A.2 Additional reduction methods for CCA

Bartlett’s test estimates how many canonical correlations are statistically significant, but its iterative computation is expensive and is not the method emphasized here.

Bartlett’s test estimates the number of statistically significant canonical correlations and averages across the corresponding important CCA vectors.The null hypothesis states that k canonical correlations are significant and the remaining correlations are random.
Under the null, Bartlett’s statistic follows a chi-squared distribution with (a−k)(b−k) degrees of freedom.Here, a and b are the neuron counts in L1 and L2, and c = min(a, b).
The method is expensive because its metric is iterative, so the work focuses on projection weighting instead.

A.3 Representation Dynamics in RNNs Through Sequence (Time) Steps

CCA reveals that RNN hidden states can remain similar under linear recurrent transformations, yet vary nonlinearly across sequence timesteps when inputs and sequence history change.

Toy RNN demonstrations: CCA recognized similarity across timesteps in linear and blended linear/nonlinear toy RNNs where cosine and Euclidean distances largely failed.The blended update included a nonlinear component scaled up to α=100.
Trained LSTM dynamics: In trained LSTMs, hidden-state representations varied significantly across the 20 unrolled sequence timesteps on PTB and WikiText-2.The analysis compared each timestep-specific activation matrix with the final timestep using CCA, cosine, and Euclidean distance.
Interpretation: CCA therefore separates linear-transform similarity from nonlinear variation that emerges with unique inputs and changing sequence history.Even CCA failed to find similarity until late in sequences containing unique inputs.
Repeated-input controls: With repeated inputs introduced early, CCA found highly similar hidden states while cosine and Euclidean distance remained insensitive.This pattern suggests that recurrent dynamics are approximately linear under early repetition.
Repeated-input controls: When identical inputs were repeated late, CCA distance increased markedly, indicating dependence on sequence history as well as the fixed recurrent matrix.The result distinguishes input variability from recurrent dynamics by holding the repeated inputs fixed while changing their position in the sequence.

A.4 Experimental details

The experiments combine fixed CIFAR-10 ConvNet and PTB/WikiText-2 LSTM setups with synthetic signal-and-noise tests for CCA-based similarity analysis.

CIFAR-10 ConvNets: CIFAR-10 ConvNets used a default 11-layer filter configuration, 3x3 kernels, batch size 32, and Adam with learning rate 0.001 for 100 epochs.The listed default filter sizes were 64, 64, 128, 128, 128, 256, 256, 256, 512, 512, 512.
RNN experiments: PTB and WikiText-2 RNN experiments used open-source word-level LSTM code with unmodified hyperparameters and no finetuning or continuous cache pointer augmentation.The setup was designed to support reproduction by training the models and applying CCA to their hidden states.
Toy experiments: Synthetic experiments generated 200-neuron representations with k standard-normal signal dimensions and 200−k lower-magnitude Gaussian noise dimensions.The signal dimension k ranged from 20 to 199, while the noise variance was 0.1.
Toy experiments: The synthetic comparison applied a random orthonormal transform to the signal portion before using CCA-based methods to detect similarity between X and Y.The experiments specifically examined cases where the signal dimension was much smaller than the total representation.

A.5 Additional control experiments

Additional controls show that CCA captures representational relationships that cosine and Euclidean distances miss, including effects of network size, training, and test-data generalization.

Generalization versus memorization: Cosine and Euclidean distances failed to reveal the distinction between converged generalizing and memorizing network solutions that CCA detected.Unweighted CCA and SVCCA recovered the broad pattern but missed that generalizing networks became more similar in the final two layers.
Network width: CCA showed that wider CIFAR-10 networks converged to more similar solutions than narrower networks, whereas cosine and Euclidean distances did not reveal this relationship.The corresponding relationship was absent at initialization and appeared after training.
RNN training dynamics: Cosine and Euclidean controls reproduced bottom-up convergence patterns in shallow PTB and WikiText-2 RNNs, supporting the training-dynamics analysis.Both metrics displayed properties of bottom-up convergence across the tested shallow RNN settings.
Test-data control: On test data, generalizing networks converged to similar softmax solutions, while memorizing networks did not.The comparison used pairwise CCA distances within each group and between generalizing and memorizing networks across layers.