PaperlayerView exact revision on arXiv

Source-linked AI summary

Questioning Representational Optimism in Deep Learning: The Fractured Entangled Representation Hypothesis

Akarsh Kumar, Jeff Clune, Joel Lehman, Kenneth O. Stanley

arXiv:2505.11581v1cs.CVcs.LGcs.NE

TL;DR

The paper questions whether scaling data and compute makes good internal representations emerge as performance improves. It compares open-ended evolved and SGD-trained networks on a single-image task. Their output behavior matches, but their internal representations differ.

  • Problem

    The paper questions whether scaling data and compute makes good internal representations emerge as performance improves.

    Source: §1 · p. 2

  • Method

    The paper compares open-ended evolved and SGD-trained networks on a single-image task, visualizing each neuron's behavior as an image.

    Source: §1 · p. 4, §2.1 · p. 5, §3 · p. 8

  • Results

    The networks produce matching output behavior but differ in their internal representations.

    Source: §1 · p. 4, §3 · p. 8

  • Takeaways & Limitations

    Benchmark performance alone can miss differences between outward behavior and internal representations.

    Source: §8 · p. 22

    GPT-4 no longer exhibits one specific FER behavior, but whether this reflects better coverage or unification remains unclear and needs further study.

    Source: §5.1 · p. 13

Abstract

from arXiv · show

Much of the excitement in modern AI is driven by the observation that scaling up existing systems leads to better performance. But does better performance necessarily imply better internal representations? While the representational optimist assumes it must, this position paper challenges that view. We compare neural networks evolved through an open-ended search process to networks trained via conventional stochastic gradient descent (SGD) on the simple task of generating a single image. This minimal setup offers a unique advantage: each hidden neuron's full functional behavior can be easily visualized as an image, thus revealing how the network's output behavior is internally constructed neuron by neuron. The result is striking: while both networks produce the same output behavior, their internal representations differ dramatically. The SGD-trained networks exhibit a form of disorganization that we term fractured entangled representation (FER). Interestingly, the evolved networks largely lack FER, even approaching a unified factored representation (UFR). In large models, FER may be degrading core model capacities like generalization, creativity, and (continual) learning. Therefore, understanding and mitigating FER could be critical to the future of representation learning.

1 Introduction

The paper questions the assumption that scaling and stronger performance naturally yield better internal representations. It introduces fractured entangled representation (FER) and contrasts conventional SGD with open-ended evolution, finding markedly different internal structures despite identical output behavior.

1 Introduction

  • The paper challenges representational optimism: improved performance with scaling does not establish that internal representations are improving.The authors argue that representation quality has received less attention as models grow.Source: §1 · p. 2
  • FER describes concepts split into disconnected, redundant mechanisms whose functions become entangled with otherwise independent behaviors.The paper contrasts this with an aspirational unified factored representation (UFR), in which capabilities are unified and kept separate; UFR need not be attainable in practice.Source: §1 · pp. 2–3
  • FER may weaken generalization in sparse-data regions by making relevant principles apply only selectively and introducing unintended side effects.The paper highlights the borderlands of human knowledge as a particularly concerning setting for such sparse coverage.Source: §1 · p. 3
  • FER may hinder creativity by impeding the extension of a concept’s regularities, and weight-sweep studies report an advantage for varying concepts in networks closer to UFR.The authors also argue that fracture may impede learning when nearby weight configurations fail to preserve regularities, with the tax potentially compounding in continual learning.Source: §1 · pp. 3–4
  • Single-image networks trained by open-ended evolution and conventional SGD can have identical output behavior but dramatically different internal representations.The image task makes intermediate representations easy to visualize, enabling neuron-by-neuron inspection.Source: §1 · p. 4
  • The authors do not claim that all SGD systems exhibit FER or propose a single solution; they aim to initiate discussion and further study.They note that FER may also arise with other methods, particularly when optimizing a single fixed objective.Source: §1 · p. 4

2 Background

The paper uses Picbreeder as a comparison point for studying how different search processes can produce different internal representations. CPPNs make this comparison visually inspectable, and Picbreeder’s open-ended process yielded networks with structured, factored representations.

2 Background

  • Picbreeder offers a comparison between open-ended search and conventional training, not a claim that its search process is superior.The paper uses this comparison to reconsider what constitutes a good representation and how it might be achieved.Source: §2 · p. 4

2.1 Compositional Pattern Producing Networks (CPPNs)

  • A CPPN maps pixel coordinates x, y, and d to hue, saturation, and value, allowing its full output behavior and each hidden neuron’s behavior to be visualized as images.CPPNs can reuse network structures across coordinates to produce patterns with regularities and symmetries.Source: §2.1 · p. 5
  • NEAT evolves CPPN weights, topology, and neuron activation functions, allowing networks to grow from smaller ones while building on earlier regularities.Picbreeder’s open-ended search supplies a point of comparison with conventional methods by producing different kinds of internal representations.Source: §2.1 · p. 6
  • A CPPN’s image represents outputs across the full input space, unlike a single instance produced by a modern generative image model.The paper treats this complete output space as a rough analogy for the range of responses of a large language model, while noting that the LLM space is much vaster.Source: §2.1 · p. 6

2.2 Picbreeder

  • Picbreeder users selected images to breed without a fixed target, enabling open-ended search, goal switching, and continued generation of novel images.The search could pivot when mutations led to an unexpected direction, such as an egg becoming a teapot.Source: §2.2 · pp. 6–7
  • Picbreeder CPPNs commonly contained factored representations that respected visual regularities, with separate controls for dimensions such as an apple’s size and stem angle.The paper describes these networks as having unified factored representations.Source: §2.2 · pp. 7–8

3 Unified Factored Representations (UFR)

The section uses the Picbreeder skull CPPN as a canonical UFR example and compares it with an SGD-trained replica. Neuron-level visualizations and weight sweeps show reuse, semantic factorization, and coherent variation in the evolved network.

3 Unified Factored Representations (UFR)

  • The Picbreeder skull provides a comparison case because supervised learning can train a CPPN to reproduce its output, enabling inspection of both networks’ representations.The SGD model uses the same architecture as the layerized Picbreeder CPPN and is trained to match its output.Source: §3 · p. 8
  • Distinct neurons represent the skull’s overall shape and features such as eyes, nose, and mouth, with symmetry emerging early and persisting across layers.The authors characterize this as a UFR in which key features appear once and influence the skull as a whole.Source: §3 · p. 9
  • The Picbreeder CPPN builds the skull through organized, reused representations, while the SGD version represents the same skull as a disorganized patchwork.Figure 5 visualizes each layer-and-neuron representation; the layerized Picbreeder CPPN has 24 novel representations.Source: §3 · p. 9, Figure 5 · p. 10
  • Sweeping Picbreeder CPPN weights produces orderly, meaningful changes that can preserve skull structure and independently control semantic features such as mouth width.The displayed sweeps were selected to present both methods in their best light while remaining broadly representative; all possible sweeps are disclosed.Source: §3 · p. 11, Figure 6 · p. 11
  • The Picbreeder representation generally preserves symmetry under weight perturbations, although the winking sweep breaks symmetry in an orderly, coherent way.The section frames underlying representation as relevant to building new behaviors, not just producing one correct image.Source: §3 · pp. 9, 11

4 Fractured Entangled Representations (FER)

The SGD-trained CPPN reproduces the Picbreeder skull but represents it through a disorganized patchwork that does not use symmetry through its hidden layers. Its weight perturbations also break symmetry incoherently, while the authors consider whether a different feature basis could explain the apparent disorder.

4 Fractured Entangled Representations (FER)

  • Despite matching the Picbreeder skull, the SGD CPPN does not encode its symmetry until the output layer across its 12 underlying layers, yielding a patchwork with many novel patterns.The authors describe this as evidence suggestive of FER.Source: §4 · p. 12
  • Weight perturbations of the SGD CPPN almost always break symmetry incoherently, unlike the Picbreeder CPPN’s usually preserved or orderly varied symmetry.The two CPPNs appear identical in their unperturbed output.Source: §4 · p. 12
  • A rotated feature basis might reveal more unified features, but the authors consider this unlikely given the disorder and arbitrary high-frequency patterns in the SGD representation.Appendix F uses PCA and basis-agnostic weight sweeps to provide additional evidence addressing this alternative.Source: §4 · p. 12
  • The authors argue that this representation limits building new skulls or faces that require understanding facial regularities, despite the model drawing a perfect skull.They also report the same general FER phenomenon in other Picbreeder examples and in standard ReLU networks trained with SGD.Source: §4 · p. 12

5 Imposter Intelligence

The skull serves as a metaphor for intelligence that appears successful across its behaviors despite fractured internal representations. The authors argue that language-model behaviors also suggest FER, while emphasizing that specific behavioral signs may disappear as models scale or receive further training.

5 Imposter Intelligence

  • The skull illustrates how a model can match expected outputs across inputs while its internal representation differs from what those outputs suggest.The paper presents CPPN images as a metaphor for a network’s broader input-output behavior, not a single generation step.Source: §5 · p. 12

5.1 Evidence of Imposter Intelligence

  • GPT-3 answered an object-counting question correctly but overcounted animals, which the authors interpret as evidence of different neural algorithms across contexts.The paper argues that summing should invoke the same algorithm regardless of the object types.Source: §5.1 · p. 13
  • GPT-4 no longer exhibits this specific counting behavior, but the authors say it is unclear whether this reflects better coverage or a more unified representation.They note that behavioral evidence of FER can become harder to find in more recent models without necessarily indicating that underlying FER has disappeared.Source: §5.1 · pp. 13–14
  • The Othello model learned local heuristics rather than unified rules, while arithmetic studies likewise report collections of context-specific heuristics.The paper suggests these may be overfit to common situations rather than fundamental rules that generalize to counterfactual contexts.Source: §5.1 · p. 14
  • GPT-4o can easily replace numbers at specified positions in a sequence, but answered word-position replacements incorrectly in all 3 of 3 responses, suggesting failure to apply the same algorithm across contexts.Inappropriate substitutions such as “were” or “are” for “am” suggest entanglement between grammatical ability and instruction following.Source: §5.1 · pp. 14–15
  • GPT-4o fails to add one thumb to an ape hand but succeeds with a human hand, suggesting the concept is not applied as a general regularity.The paper also notes that the ape-hand failures vary across attempts.Source: §5.1 · p. 15, Figure 7 · p. 16
  • The authors conclude that LLM behaviors suggest FER, and that SGD tends to produce FER in their experiments without special intervention.They argue that fixing an individual error would not establish that FER has disappeared from the underlying representation.Source: §5.1 · p. 16

6 Key Factors in FER and UFR

The discussion considers how learning order, network growth, data, architecture, and search may shape whether representations become fractured or unified. It presents open-ended search as a promising route, while emphasizing that the mechanisms and effective mitigations remain uncertain.

6.1 The Role of Order and the Ability to Reorganize Representation

  • Learning order can shape eventual representations: Picbreeder’s skull inherited symmetry discovered earlier, while conventional SGD reached the skull without first building such regularities.The paper argues that learning concepts in parallel when they depend on one another may encourage separate heuristics; intentional human reanalysis has no clear current analog in LLMs.Source: §6.1 · pp. 17–19

6.2 NEAT and Increasing Complexity

  • NEAT’s gradual growth from simple networks may help Picbreeder discover organizing principles early, but open-ended search may matter more than NEAT itself.The paper notes that objective-driven NEAT produced more bloated representations than Picbreeder’s discoveries, and that Picbreeder demonstrates UFR is possible in neural networks.Source: §6.2 · p. 19

6.3 More Data and Holistic Unification

  • More examples might unify representations, as the paper suggests for GPT-4 arithmetic, but they could also yield more fractured representations rather than holistic unification.Whether grokking can clean up FER remains open; its dependence on parameterization and dataset size complicates assuming it will occur reliably, while frontier knowledge remains sparse.Source: §6.3 · pp. 19–20

6.4 Architectural and Algorithmic Changes

  • Architectural or algorithmic changes might mitigate FER, but their effects remain uncertain and require rigorous empirical study.The paper notes that architectures differ in weight sharing and sparsity, which could affect their prospects for mitigating or exacerbating FER.Source: §6.4 · pp. 20–21

6.5 Open-Ended Search

  • Open-ended search may foster modularity, hierarchy, and UFR through serendipitous stepping stones, but no definitive process is yet paired with SGD for large models.Picbreeder’s early symmetry discovery became an organizing basis for later discoveries; the paper identifies MAP-Elites, MCC, POET, and OMNI as possible inspiration.Source: §6.5 · pp. 21–22, Figure 10 · p. 21

7 Discussion

The discussion argues that good representations must capture regularities, and that representation quality is not reducible to compression alone. The paper presents its examples as preliminary and as a starting point for broader discussion.

7 Discussion

  • Generalization, creativity, and learning depend on representations capturing regularities; missing them can degrade generalization, thwart creative ideation, and make learning expensive.The paper frames this concern around whether representations support extending, modifying, and combining what has been learned.Source: §7 · p. 22
  • Unified factored representation is distinct from compression alone because factoring can preserve dynamic features needed for creativity and continual learning.A maximally compressed skull representation might omit features such as eye winking or jaw widening; the paper argues factorization could be reached without extensive representative examples.Source: §7 · p. 22
  • The paper’s simple demonstrations do not fully illuminate these implications and are intended to start a larger discussion.Source: §7 · p. 22

8 Conclusion

The conclusion argues that outward performance does not establish that a model has a good internal representation, and that the possibility of FER in scaled systems merits scrutiny. The appendix describes NEAT as an evolutionary algorithm that changes both network weights and topology.

8 Conclusion

  • Benchmark performance may miss differences between internal representations, and the hope that FER vanishes with scale merits further scrutiny.The paper says opinions may vary about FER’s seriousness and presents its claims as a reason for additional research.Source: §8 · p. 22

A.1 Details of NEAT

  • NEAT evolves both network weights and topology, beginning with simple networks that become more complex over generations.It encodes networks as directed acyclic graphs of node and connection genes, and innovation numbers support historical tracking and genome alignment during crossover.Source: §A.1 · p. 29
  • Picbreeder omits NEAT’s speciation feature because human selection across many objectives supplies an implicit diversity pressure.Source: §A.1 · p. 29

A.2 Details of CPPNs

The appendix explains how CPPNs map coordinate inputs to image pixels and how Picbreeder users evolve and share those networks. It also notes that the paper provides data and code for three example CPPNs.

A.2 Details of CPPNs

  • A CPPN receives pixel coordinates and additional inputs, produces raw HSV values, and converts them to RGB for image generation.Sweeping over the x-y grid allows the process to generate an image at any resolution.Source: §A.2 · p. 29

A.3 Details of Picbreeder

  • Picbreeder users select preferred CPPN images from each generation, which are mutated or crossed over, and published images can seed later users’ searches.Source: §A.3 · p. 29
  • The paper provides data for its skull, butterfly, and apple CPPNs, along with code to interpret it.Source: §A.3 · p. 29

B Porting NEAT CPPNs to Dense MLP CPPNs

NEAT CPPNs use irregular graph-like connections that differ from dense, sequential MLPs. Layerization converts a NEAT CPPN into an MLP that preserves its computation while enabling SGD optimization.

B Porting NEAT CPPNs to Dense MLP CPPNs

C SGD Training Details

SGD-trained CPPNs are initialized with LeCun normal weights and trained to reproduce 256 × 256 Picbreeder images using full-batch MSE.

C SGD Training Details

  • The CPPNs are trained on 256 × 256 target images for 100,000 iterations using full-batch MSE and a learning rate of 3 × 10−3.The apple CPPN uses a learning rate of 3 × 10−4; weights are initialized with LeCun normal initialization.Source: C SGD Training Details · p. 31
  • LeCun normal initialization is used for the SGD-trained CPPN weights.Source: C SGD Training Details · p. 31
  • The described training setup reproduces the Picbreeder images without difficulty.Source: C SGD Training Details · p. 31

D Other CPPN Examples

Butterfly and apple CPPNs from Picbreeder show more factored internal representations than their conventional SGD counterparts, despite matching the target images.

D Other CPPN Examples

  • The SGD-trained butterfly CPPN reconstructs the image but exhibits FER, while the Picbreeder CPPN shows symmetry and regular structure supported by UFR.Source: D Other CPPN Examples · p. 31
  • Picbreeder’s butterfly weight sweeps preserve symmetry while making meaningful semantic changes; SGD’s frequently breaks symmetry and introduces deleterious changes.The reported trajectories through image space also differ between Picbreeder and conventional SGD.Source: D Other CPPN Examples · p. 32
  • The Picbreeder apple CPPN factors background, body, and stem, whereas the SGD version models stem and body together and often entangles multiple concepts in its weight sweeps.A Picbreeder weight can control the stem’s angle independently.Source: D Other CPPN Examples · p. 32

E ReLU CPPNs

ReLU-only MLPs are used to test whether FER also appears with more conventional activation functions, and their visualized representations exhibit FER.

E ReLU CPPNs

  • ReLU-only MLPs are trained to test whether FER also occurs with conventional activation functions.Source: E ReLU CPPNs · p. 32
  • The visualized intermediate representations of the ReLU-only MLPs for the skull, butterfly, and apple all exhibit FER.Source: E ReLU CPPNs · p. 32

F Change of Basis of the Feature Space

The section tests whether SGD’s apparent representational disorganization is merely a basis-dependent view. PCA improves organization only slightly, while weight sweeps agnostic to orthonormal transformations still reveal a contrast between Picbreeder and SGD representations.

F Change of Basis of the Feature Space

  • The Picbreeder butterfly representation develops y-axis symmetry early and specifies each output aspect once, whereas SGD separately encodes the wings amid stitched high-frequency patterns.The paper characterizes these as UFR and FER, respectively.Source: Figure 14 · p. 33, Figure 15 · p. 34
  • PCA projects layer feature data onto principal components for visualization, improving SGD skull organization slightly but remaining far from Picbreeder’s UFR.PCA is one possible transformation; the section notes that exhaustive evaluation would require considering other orthonormal transformations.Source: F Change of Basis of the Feature Space · p. 33, Figure 24 · p. 42
  • For skull CPPNs, random-vector column sweeps preserve meaningful, y-axis-symmetric Picbreeder changes but yield meaningless SGD distortions; butterfly sweeps likewise show an organizational contrast.The sweeps are designed to be agnostic to orthonormal transformations of the weight matrix.Source: F Change of Basis of the Feature Space · p. 34, Figure 25 · p. 43
  • Butterfly weight sweeps preserve symmetry for Picbreeder but break it for SGD; the discovery paths show Picbreeder building on symmetry while SGD greedily optimizes without recognizing the wings’ connection.The Picbreeder path proceeds from symmetry to structure and then color.Source: Figure 16 · p. 35, Figure 17 · p. 35
  • Picbreeder’s apple representation separately encodes stem and body before combining them, while SGD entangles nearly every neuron without clear separation of apple, stem, and background.Only a small number of Picbreeder apple feature maps are described as genuinely novel.Source: Figure 18 · p. 36, Figure 19 · p. 37
  • Apple weight sweeps produce human-meaningful Picbreeder changes, while SGD sweeps couple features, such as changing the stem with apple size or changing stem color with background color.The coupled changes suggest inherent entanglement in the SGD representation.Source: Figure 18 · p. 36, Figure 20 · p. 38
  • SGD also shows FER with ReLU networks for skulls, butterflies, and apples; in the skull, high-frequency artifacts cancel at the output but still degrade the internal representation.The ReLU skull feature maps contain skull shapes with erratic high-frequency patterns added.Source: Figure 21 · p. 39, Figure 22 · p. 40, Figure 23 · p. 41

Related papers

Loading 2505.11581v1…