Source-linked AI summary
Convergent Learning: Do different neural networks learn the same representations?
Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, John Hopcroft
TL;DR
The paper asks whether independently trained neural networks learn convergent internal representations, a question that matters for understanding and improving deep models. It compares representations across networks using bipartite matching, sparse prediction, and spectral clustering, finding shared features and subspaces alongside inconsistent features and non-identical basis vectors.
Problem
The paper investigates whether independently trained neural networks learn largely similar features or radically different representations, addressing a difficult but important problem in understanding deep models.
Method
The paper trains multiple networks and approximately aligns their representations using one-to-one bipartite matching, sparse prediction, and many-to-many spectral clustering.
Results
Some features are learned reliably across networks, while others are inconsistent; shared low-dimensional subspaces can have different basis vectors, and codes mix local with slightly distributed representations.
Takeaways & Limitations
Convergent learning is partial: networks share core features and subspaces without learning identical representations.
Takeaways & Limitations
Unique one-to-one matching can fail when networks span the same subspace with different numbers of units, and more powerful predictive models may require other hyperparameters or architectures.
Abstract
from arXiv · showhide
Recent success in training deep neural networks have prompted active investigation into the features learned on their intermediate layers. Such research is difficult because it requires making sense of non-linear computations performed by millions of parameters, but valuable because it increases our ability to understand current models and create improved versions of them. In this paper we investigate the extent to which neural networks exhibit what we call convergent learning, which is when the representations learned by multiple nets converge to a set of features which are either individually similar between networks or where subsets of features span similar low-dimensional spaces. We propose a specific method of probing representations: training multiple networks and then comparing and contrasting their individual, learned representations at the level of neurons or groups of neurons. We begin research into this question using three techniques to approximately align different neural networks on a feature level: a bipartite matching approach that makes one-to-one assignments between neurons, a sparse prediction approach that finds one-to-many mappings, and a spectral clustering approach that finds many-to-many mappings. This initial investigation reveals a few previously unknown properties of neural networks, and we argue that future research into the question of convergent learning will yield many more. The insights described here include (1) that some features are learned reliably in multiple networks, yet other features are not consistently learned; (2) that units learn to span low-dimensional subspaces and, while these subspaces are common to multiple networks, the specific basis vectors learned are not; (3) that the representation codes show evidence of being a mix between a local code and slightly, but not fully, distributed codes across multiple units.
1 INTRODUCTION
The paper asks whether independently trained neural networks learn radically different features or exhibit convergent learning, where representations are similar individually or span similar subspaces. It begins addressing this question by comparing learned representations across networks using multiple alignment approaches.
- Motivation: The central question is whether networks trained on the same task learn unique feature sets or largely convergent representations.The question matters for understanding neural-network training and potentially designing better architectures, learning algorithms, and hyperparameters.
- Approach: The study trains multiple same-architecture networks from different random initializations and compares their learned representations.The networks use an AlexNet-derived architecture and are trained on ImageNet.
- Research questions: The paper investigates one-to-one neuron alignment, asking whether units in one network have close counterparts in another or whether representations are unique.It also asks whether alignment is robust to correlation and estimated mutual-information similarity measures.
- Research questions: The paper further asks whether imperfect one-to-one alignment reflects rotated or affine-transformed representation spaces, motivating sparse predictive mappings.The stated answer is that a shared core representation exists, but some rare features appear in only one network.
2 EXPERIMENTAL SETUP
The experiments compare activation statistics and correlations across four independently initialized AlexNet-derived networks trained on ImageNet. Correlation matrices quantify within-network and between-network unit similarity at corresponding layers.
- Architecture: The networks use five convolutional layers followed by three fully connected layers, with full connectivity replacing AlexNet’s artificial grouped structure.The layers are conv1–conv5 followed by fc6–fc8.
- Visualization: Figure 1 contrasts within-network and between-network correlation matrices before and after permuting one network to approximate the other’s feature order.The partially white diagonal in the permuted matrix indicates alignment success.
- Training: Four networks trained from different random initializations achieve top-1 accuracies of 58.65%, 58.73%, 58.79%, and 58.84%.These results are close to the original study’s reported 59.3% top-1 accuracy.
- Activation statistics: Activation statistics are aggregated over the ILSVRC validation set and, for convolutional layers, across spatial positions.The collected activations cover units in conv1–conv5 and fc6–fc7.
- Similarity measure: Within-network correlations form symmetric square matrices whose side length equals the number of units in the layer.For example, conv1 uses a 96 × 96 matrix.
- Similarity measure: Correlation measures activation similarity within a network or between networks while remaining independent of activation scale.Only units on the same layer are compared, and mutual information produces similar results to correlation.
3 IS THERE A ONE-TO-ONE ALIGNMENT BETWEEN FEATURES LEARNED BY DIFFERENT NEURAL NETWORKS?
The paper tests whether independently trained networks admit one-to-one feature alignments, using correlation and mutual-information-based assignments. Many units align closely, but alignment quality varies by layer and some features appear network-specific.
- Correlation-based alignment: Correlation matrices are used to align units across networks through semi-matching and one-to-one bipartite matching.Semi-matching permits multiple Net1 units to share a Net2 match, whereas matching enforces unique pairings.
- Correlation-based alignment: Many units have unique, highly correlated counterparts after matching, indicating substantial shared structure between networks.The permuted between-network correlation matrix has many bright diagonal entries, although they are generally below the self-correlation value of one.
- Layer-wise convergence: Alignment is strongest in conv1 and conv5 and weaker in intermediate layers, with the paper describing conv1 and conv2 as more convergent than conv3 and conv4.The reported layer pattern makes the intermediate region the most difficult part of the alignment path.
- Matching limitations: Unique matching can perform much worse when networks use different numbers of units to span the same subspace.A surplus unit may be forced into an almost unrelated pairing when the other network represents that subspace with fewer filters.
- Rare and network-specific features: Semi-matching correlations decline from near 1 for the best pairs to near 0.3 for the weakest, suggesting that some filters are rarely learned across networks.The paper reports this pattern across conv1–conv5 and interprets weakly matched filters as network-specific or lacking a high-correlation counterpart.
- Mutual-information check: Mutual-information matching produces qualitatively similar results to correlation-based matching, including seven of eight unchanged best conv1 pairs.This supports correlation as an adequate similarity measure for the conclusions presented.
4 RELAXING THE ONE-TO-ONE CONSTRAINT TO FIND SPARSE, FEW-TO-ONE MAPPINGS
One-to-one matching misses relationships between some neurons, so the paper relaxes the constraint by learning sparse mappings between corresponding layers. Sparse mappings predict early-layer activations accurately with only a few source units, while higher-layer results remain inconclusive.
- Motivation: 17% of conv1 neurons, 37% of conv2, 63% of conv3, 92% of conv4, and 75% of conv5 neurons lack a cross-network match above correlation 0.5.These results show that one-to-one correspondence becomes especially insufficient in intermediate layers.
- Sparse mappings: An L1-penalized mapping relaxes permutation matching by learning few-to-one or one-to-one correspondences, with stronger penalties producing sparser weights.The penalty can range from mostly one-to-one mappings to dense linear mappings when set to zero.
- Results: For conv1 and conv2, prediction error remains nearly unchanged from dense mappings until the L1 penalty exceeds 10^-3.The representations are normalized so channels receive equal importance and prediction errors are roughly comparable across layers.
- Results: A conv1 mapping with penalty 10^-2.6 achieves sparse prediction loss 0.235 using an average of 4.7 source units per target unit, while conv2 uses 2.5 connections per target unit at 10^-3.Higher layers conv3–conv5 perform poorly even without regularization, so they are excluded from further results.
- Interpretation: Together, one-to-one and sparse mappings suggest a representation code that mixes local features with slightly distributed codes rather than being fully local or fully distributed.Well-matched units support local coding, whereas sparsely predictable but unmatched units support slightly distributed coding.
- Shared subspaces: Sparse prediction indicates that small groups of units span similar subspaces, with hierarchical clustering exposing both predictive unit pairs and larger cross-network clusters.The clustering starts from a block matrix built from identity and learned mapping weights, then greedily merges the strongest relationships.
5 CONCLUSIONS
The paper quantifies similarities between independently trained networks by aligning neurons and subspaces, revealing shared features alongside network-specific variation. Its findings indicate mixed local and mildly distributed codes, common subspaces with differing bases, and convergent activation distributions.
- The paper blends bipartite matching, sparse prediction and clustering to align independently trained networks at feature and subspace levels.
- Some features are learned reliably across networks, while other features are not consistently learned.
- Networks share low-dimensional subspaces, but the specific basis vectors spanning those subspaces differ.
- Representation codes mix single-unit local coding with slightly, but not fully, distributed coding across multiple units.
- Average neuron activations vary considerably within a network, while mean activation distributions converge across networks.
- HAC results show many cross-network one-to-one leaf clusters and higher-level clusters forming shared multi-unit subspaces.
6 FUTURE WORK
The paper identifies model compression and ensemble formation as future directions based on shared and rare features across independently trained networks.
- Future work could test whether removing low-correlation, rare filters affects model performance.
- Future work could examine how feature correlation among independently trained networks affects ensemble performance.
S.1 ACTIVATION VALUES: HIGH CORRELATION VS. LOW CORRELATION
Activation values for conv1 and conv2 are measured across 5,000 randomly sampled images and all spatial positions. The resulting joint distributions appear simple enough that correlation can identify matching units.
- 5,000 randomly sampled images provide the activation measurements for conv1 and conv2.
- Activation values are collected across all spatial positions: 55 × 55 in conv1 and 27 × 27 in conv2.
- The joint distributions appear sufficiently simple to suggest that correlation is adequate for finding matching units.
S.2 ADDITIONAL MATCHING RESULTS
Additional figures compare semi-matching and matching assignments across convolutional layers and visualize the strongest and weakest aligned features. The comparisons use correlation and mutual information alongside activation-based visualizations.
- Additional matching results: Semi-matching and matching assignments are compared for units in conv2–conv5 by plotting both correlation values after sorting by semi-matching value.
- Additional matching results: Figure S2 visualizes the eight best and eight worst correlated matched features across conv1–conv5 using highly activating validation patches and deconv visualizations.
- Additional matching results: Figure S3 encodes semi-matching pairings with large light green circles and matching pairings with small dark green dots.
- Additional matching results: Figure S4 presents the eight highest- and eight lowest-mutual-information features from semi-matching in conv1 and conv2.
S.3 DOES RELAXING THE ONE-TO-ONE CONSTRAINT TO FIND MANY-TO-MANY GROUPINGS REVEAL MORE SIMILARITIES BETWEEN WHAT DIFFERENT NETWORKS LEARN?
Because neurons may not align globally one-to-one across networks, the paper seeks many-to-many matchings using spectral clustering.
- Spectral clustering is used to find many-to-many correspondences between neurons in different networks.
S.3.1 NEURON SIMILARITY GRAPHS
The paper represents within- and between-network activation relationships as similarity graphs, then combines them to identify groups of related neurons.
- Three similarity graphs are defined from correlation matrices: single-network, between-network, and combined graphs.
- Single-net neuron similarity graphs: In single-network graphs, units are vertices connected when their activation correlation exceeds threshold τ.
- Between-net neuron similarity graphs: Between-network graphs are bipartite and connect units from different networks when their cross-network correlation exceeds the threshold.
- Combined similarity graphs: The combined graph partitions neurons so within-group edges are relatively strong and between-group edges are very weak.
S.3.2 SPECTRAL CLUSTERING AND METRICS FOR NEURON CLUSTERS
Spectral clustering organizes correlated neurons into cross-network groups, revealing shared feature structure alongside differences in activation magnitude and feature ordering.
- Spectral clustering: The combined correlation matrix W_l and unnormalized Laplacian L_l = D_l − W_l support spectral clustering into neuron groups.The first k eigenvectors associated with the smallest eigenvalues provide the lower-dimensional embedding used for clustering.
- Spectral clustering results: Many-to-many feature-map correspondences appear across networks, including clusters with similar neurons from both Net1 and Net2.The figures show clustered matchings across convolutional layers.
- Spectral clustering results: Conv1 matching clusters include Gabor filters and color blobs, indicating that several general low-level features recur across networks.
- Hierarchical clustering: Hierarchical refinement produces leaf clusters containing fewer than 6 neurons from each network when α = 0.025 in conv2.The refinement recursively reclusters groups larger than 2α · S_l.
- Hierarchical clustering: Conv2 units {62, 137, 148} from Net1 and {33, 64, 230} from Net2 learn similar red and magenta object features.
- Activation magnitudes: Across four networks, mean activations vary by one or two orders of magnitude within layers, while their overall activation spectra roughly correspond.The highest-average conv2 activation ranges from 49 to 120, and the conv1 range is 98 to 130.
- Activation ordering: The ordering of highly active conv1 filters differs across networks: color patches and edge orientations do not consistently occupy the top ranks.