PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Understanding Learning Representations: To What Extent Do Different Neural Networks Learn the Same Representation

Liwei Wang, Lunjia Hu, Jiayuan Gu, Yue Wu, Zhiqiang Hu, Kun He, John Hopcroft

arXiv:1810.11750v2cs.LGstat.ML

TL;DR

The paper asks how similar representations are across identically structured networks trained from different initializations, a question for which systematic characterization is lacking. It develops a theory and algorithms for activation-subspace matches, and its experiments find low similarity in most convolutional layers. The result leaves open whether this reflects different learned representations or a limitation of subspace match as a metric.

  • Problem

    Despite the perceived importance of learned representations, systematic quantitative characterization is lacking, including evidence about similarity across differently initialized networks and its local or distributed structure.

    Source: §1 · pp. 1–2

  • Method

    The paper models neurons by activation vectors and characterizes similarity through activation-subspace matches, deriving maximum and simple matches and algorithms to find them.

    Source: §1 · p. 2, §2 · p. 3

  • Results

    Most convolutional layers exhibit very low similarity in terms of subspace matches, contrary to prevalent expectations.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The finding leaves open whether differently initialized networks learn different representations or whether subspace match is an inadequate similarity metric.

    Source: §6 · p. 9

    The reported similarity conclusion is scoped to the subspace-match measure and does not resolve whether the representations themselves differ.

    Source: §1 · p. 2, §6 · p. 9

Abstract

from arXiv · show

It is widely believed that learning good representations is one of the main reasons for the success of deep neural networks. Although highly intuitive, there is a lack of theory and systematic approach quantitatively characterizing what representations do deep neural networks learn. In this work, we move a tiny step towards a theory and better understanding of the representations. Specifically, we study a simpler problem: How similar are the representations learned by two networks with identical architecture but trained from different initializations. We develop a rigorous theory based on the neuron activation subspace match model. The theory gives a complete characterization of the structure of neuron activation subspace matches, where the core concepts are maximum match and simple match which describe the overall and the finest similarity between sets of neurons in two networks respectively. We also propose efficient algorithms to find the maximum match and simple matches. Finally, we conduct extensive experiments using our algorithms. Experimental results suggest that, surprisingly, representations learned by the same convolutional layers of networks trained from different initializations are not as similar as prevalently expected, at least in terms of subspace match.

1 Introduction

The paper addresses the gap between intuition about learned representations and a systematic way to characterize them, asking how similar representations are across identically structured networks trained from different initializations. It develops a theory of activation-subspace matches and reports low similarity in most convolutional layers.

1 Introduction

  • Similar performance after different initializations [Dauphin et al., 2014] motivates asking whether the networks also learn similar representations and whether similarity is local or distributed.The paper connects these questions to the debate over local versus distributed representations.Source: §1 · p. 2
  • The paper characterizes neuron and cluster similarity by matching subspaces spanned by neuron activation vectors.Activation vectors represent neurons as input-output functions and account for the input distribution; clusters are represented by the subspaces they span.Source: §1 · p. 2
  • Li et al. [2016] studied one-to-one, one-to-many, and many-to-many neuron correspondences empirically, using heuristic techniques especially for the latter mapping types.The paper motivates a more systematic theoretical investigation of these questions.Source: §1 · p. 2
  • The theory characterizes match structure through a unique maximum match and simple matches that cannot be decomposed further.The maximum match captures overall similarity, while simple matches are its finest units; the paper proposes efficient algorithms to find both.Source: §1 · p. 2
  • Most convolutional layers show very low representation similarity across initializations when measured by subspace matches.The authors describe this as contrary to prevalent expectations.Source: §1 · p. 2

2 Preliminaries

The model represents each neuron by its activation vector over a finite input set and each neuron subset by the span of its activation vectors. It defines approximate matches by reciprocal proximity of the two sets of vectors to the other set's span.

2 Preliminaries

  • A neuron's representation is its activation vector, containing its outputs over the d input examples.For a neuron subset, the model uses the subspace spanned by its activation vectors; the empty subset spans the zero vector.Source: §2 · p. 3
  • The subspace representation reflects that following-layer activations depend on affine combinations of the preceding neurons' activations.When two subsets span the same subspace, the prose states that following neurons can receive the same information from either subset.Source: §2 · p. 3
  • An ε-approximate match requires each neuron's activation vector in either subset to lie within ε times its norm of the other subset's span.Distance is measured using the L2 norm; an exact match is the ε = 0 case, equivalently equality of the two spans.Source: §2 · p. 3

3 A Theory of Neuron Activation Subspace Match

The theory characterizes matches through a unique maximum match and decompositions into simple matches, with different structural characterizations for exact and approximate matches. Exact matches admit unique neuron-minimal units under linear independence, while approximate matches can have multiple minimal units and need not be closed under intersection.

3 A Theory of Neuron Activation Subspace Match

  • Closure of matches under union guarantees a unique maximum match, which is the union of all matches.The maximum match captures overall similarity; simple matches indicate its finer units.Source: §3 · pp. 3–4
  • Every match decomposes into simple matches, the atomic matches that cannot be decomposed further.The decomposition need not be unique, and the number of matches can be exponential even though simple matches are polynomially bounded in specified settings.Source: §3 · p. 4, §3.2 · p. 5

3.1 Structure of Exact Matches

  • With linearly independent activation vectors, exact matches are closed under intersection, and each neuron in the maximum match has a unique v-minimum match.The simple exact matches are exactly these v-minimum matches, and their number is at most linear in the number of neurons; the paper says they can be found in polynomial time.Source: §3.1 · p. 4

3.2 Structure of Approximate Matches

  • For approximate matches, intersection closure can fail, so a neuron can have multiple v-minimal matches rather than one v-minimum match.Every v-minimal match is simple, and every simple match is a v-minimal match for some neuron in the maximum match.Source: §3.2 · p. 5
  • The v-minimal characterization requires considering neurons from either network; restricting the choice to one network does not capture all simple matches.Approximate-match decompositions into simple matches are not necessarily unique, and match differences need not remain matches.Source: §3.2 · p. 5

4 Algorithms

The algorithms find a maximum match and use neuron-specific minimal matches to recover all simple matches. Their runtime guarantees range from polynomial time for the maximum and individual minimal-match algorithms to conditional polynomial time for enumerating all simple matches.

4 Algorithms

  • Algorithm 1 outputs the maximum match in polynomial time.It starts with both neuron sets and repeatedly removes neurons whose activation vectors cannot be expressed within error ϵ by the span of the other set.Source: §4 · pp. 5–6
  • Algorithm 2 outputs one v-minimal match in polynomial time, and for exact matches with linearly independent activation vectors it returns the unique v-minimum match.It reduces the match while retaining the given neuron v.Source: §4 · p. 6
  • Algorithm 3 finds all v-minimal matches in time L O(Nv), where L is input size and Nv is their number for neuron v.It calls Algorithm 2 repeatedly and deletes a neuron from each previously found match to avoid duplicates.Source: §4 · pp. 6–7
  • Algorithm 3 is not polynomial-time in general, but strong linear independence and (ϵ, 2 sin θ + 1)-stability make Nv ≤ 1 and yield polynomial-time enumeration.Exploring all v ∈ X ∪ Y then finds all simple matches.Source: §4 · p. 7

5 Experiments

Across tested architectures and datasets, most convolutional layers have low maximum-match similarity, while simple matches reveal few small local correspondences. The experiments compare differently initialized networks and examine both overall similarity and the sizes of its finer-grained matches.

5 Experiments

  • The experiments compare VGG and ResNet networks on CIFAR10 and ImageNet, using networks trained from different random seeds and focusing on ReLU-activated neurons.Convolutional activation vectors concatenate vectorized feature maps over validation images; maximum-match similarity averages repeated random samples of outputs.Source: §5 · p. 7, §5.1 · p. 7

5.2 Simple Match

  • Maximum matching measures overall similarity, while the simple-match size distribution is used to examine finer structure.For the simple-match estimates, the experiments run randomized Algorithm 2 over neurons for several iterations and remove duplicate matches.Source: §5.1 · p. 7, §5.2 · p. 8

5.1 Maximum Match

  • Most convolutional layers have very low maximum-match similarity, nearly zero in deep networks; layers near input or output can show higher similarity under the stated alignment and dimensionality conditions.Near-output similarity is sometimes absent in the lower-accuracy VGG16/ImageNet experiment.Source: §5.1 · pp. 7–8, Figure 1 · p. 8

5.2 Simple Match

  • Even where layers near the output are similar overall, they have few small simple matches; layers near the input also have lower fine-grained similarity and few small matches.The authors conclude that almost no single neuron or small neuron set learns similar representations, even near input or output.Source: §5.2 · p. 8, Figure 2 · p. 9

6 Conclusion

The paper studies representation similarity between identically architected networks trained from different initializations, develops a theory and algorithms for subspace matches, and finds convolutional-layer representations less similar than expected. It leaves open whether this reflects genuinely different representations or a mismatch between subspace match and similarity.

6 Conclusion

  • The paper develops a rigorous theory and efficient algorithms for comparing representations learned by identically architected networks trained from different initializations.Its experiments find that convolutional-layer representations are not as similar as prevalently expected.Source: §6 · p. 9
  • Experiments find that representations learned by convolutional layers are not as similar as prevalently expected.The conclusion does not establish whether the gap reflects different representations or limitations of subspace match as a similarity metric.Source: §6 · p. 9
  • The findings leave open whether networks learn completely different representations or whether subspace match is not a good similarity metric.The paper frames resolving this distinction as a fundamental problem for deep learning and interpretability.Source: §6 · p. 9

A Omitted Proofs in Section 3

The omitted proofs establish how matches relate to simple matches and identify closure properties under union and intersection. They also distinguish the general characterization from a special exact-match result under linear-independence assumptions.

A Omitted Proofs in Section 3

  • The union of two approximate matches is still an approximate match.Source: A Omitted Proofs in Section 3 · p. 10
  • Every match can be expressed as a union of simple matches, which cannot be decomposed further.The decomposition need not be unique, as illustrated by a simple match that is a proper subset of another simple match.Source: A Omitted Proofs in Section 3 · pp. 10–11
  • Under linear independence, the intersection of two exact matches is an exact match, using the corresponding span-intersection identity.Source: A Omitted Proofs in Section 3 · p. 10
  • For any neuron in the maximum match, every v-minimal match is simple, and every simple match is v-minimal for some neuron in that maximum match.Under exact matching and linear independence, the v-minimum match gives the corresponding special case.Source: A Omitted Proofs in Section 3 · p. 10
  • The characterization restricts the v-indexed simple-match result to neurons in the maximum match; the exact-match setting permits the stated replacement under linear independence.Source: A Omitted Proofs in Section 3 · p. 11

B Omitted Proofs in Section 4

The omitted proofs establish polynomial-time procedures for the maximum match and for finding minimal and simple matches. They also give conditions under which the number of v-minimal matches is bounded, and show an intersection property for approximate matches under stability assumptions.

B Omitted Proofs in Section 4

C Complicated Aspects of the Structure of Matches

Matches have structural properties that distinguish them from ordinary set systems: they are not closed under difference, and simple matches need not form unique or non-overlapping decompositions.

C Complicated Aspects of the Structure of Matches

D Instances without Strong Linear Independence or Stability

The section shows that neither strong linear independence nor stability can be removed from Theorem 14’s assumptions. Counterexamples produce many simple matches when either condition is absent, including under orthogonality for the stability case.

D Instances without Strong Linear Independence or Stability

  • Together, the two counterexamples show that neither assumption in Theorem 14 can be removed.The first counterexample applies at ε = 0; the second satisfies π/2-strong linear independence.Source: D Instances without Strong Linear… · pp. 12, 14
  • Without strong linear independence, instances can have superpolynomially many simple exact matches spanning more than polynomially many distinct subspaces.The section states this counterexample holds even for ε = 0, when the stability assumption is trivial.Source: D Instances without Strong Linear… · pp. 12–13
  • Even with π/2-strong linear independence, dropping stability permits instances with exponentially many simple ε-approximate matches.This counterexample shows that strong linear independence alone does not make the stability assumption dispensable.Source: D Instances without Strong Linear… · pp. 12, 14

E.1 Different Architectures

Experiments on additional network architectures show maximum-match similarities like those reported earlier, suggesting that the findings may extend to most modern deep networks. The accompanying table describes architectures and validation accuracy, omitting fully connected layers.

E.1 Different Architectures

  • VGG13 and ResNet10 show maximum matching similarities like those reported earlier, suggesting the conclusions may apply to most modern deep networks.Figure 3 presents layerwise similarities under various ε values.Source: §E.1 · p. 15, Figure 3 · p. 15
  • Figure 3 covers maximum matching similarities across layers and architectures under various ε values.The passage identifies VGG13 and ResNet10 as the architectures shown.Source: Figure 3 · p. 15, §E.1 · p. 15

E.2 Details of Architecture

  • Table 1 reports architecture structure and validation accuracy, with fully connected layers omitted.Source: Table 1 · p. 16

E.3 Max Match during Training

During training, untrained networks show some similarity, while trained and untrained networks differ much more. The neuron visualizations show selected small simple matches in two fully connected layers.

E.3 Max Match during Training

  • Untrained networks show some similarity, whereas an untrained and a trained network are much more different.Figure 4 compares two untrained networks in panel (a) with the same network at different stages in panel (b).Source: §E.3 · p. 16, Figure 4 · p. 16

E.4 Neuron Visualization

  • The experiments find a few small simple matches in fully connected layers and visualize selected examples.Figures 5 and 6 show the top nine images maximizing each neuron's activation in the displayed matches.Source: §E.4 · p. 16, Figure 5 · p. 17, Figure 6 · p. 17
  • Figure 5 presents fc2 neurons and Figure 6 presents fc1 neurons, with each network shown on its own side.Each row contains the nine images that maximize one neuron's activation.Source: Figure 5 · p. 17, Figure 6 · p. 17

Related papers

Loading 1810.11750v2…