PaperlayerView exact revision on arXiv

Source-linked AI summary

Learned feature representations are biased by complexity, learning order, position, and more

Andrew Kyle Lampinen, Stephanie C. Y. Chan, Katherine Hermann

arXiv:2405.05847v3cs.LGcs.CV

TL;DR

The paper examines how learned representations relate to computation, a link that matters because representations are used to understand or improve systems. It trains models on multiple features while varying feature and learning properties, and finds that those properties bias representations even when features are learned. These biases may complicate interpretation and comparisons between systems, while the study's synthetic datasets limit how directly its findings extend to more realistic, entangled features.

  • Problem

    Representations are used to understand or improve computation, but the relationship between representation and computation is nontrivial and its linking assumptions need empirical examination.

    Source: §1 · p. 2

  • Method

    The paper characterizes representations of models trained to compute multiple input features and varies feature properties and learning conditions.

    Source: §2 · p. 3, §8 · p. 19

  • Results

    Representations are biased by feature complexity, learning order, prevalence, and output position, even after all features have been learned.

    Source: §8 · p. 19, §9 · p. 22

  • Takeaways & Limitations

    These biases may complicate interpreting learned representations or comparing them across systems in machine learning, cognitive science, and neuroscience.

    Source: §1 · p. 3, §9 · p. 22

    The study focuses on simple synthetic datasets, so biases may differ on more realistic datasets with larger sets of entangled features.

    Source: §8.1 · p. 21

Abstract

from arXiv · show

Representation learning, and interpreting learned representations, are key areas of focus in machine learning and neuroscience. Both fields generally use representations as a means to understand or improve a system's computations. In this work, however, we explore surprising dissociations between representation and computation that may pose challenges for such efforts. We create datasets in which we attempt to match the computational role that different features play, while manipulating other properties of the features or the data. We train various deep learning architectures to compute these multiple abstract features about their inputs. We find that their learned feature representations are systematically biased towards representing some features more strongly than others, depending upon extraneous properties such as feature complexity, the order in which features are learned, and the distribution of features over the inputs. For example, features that are simpler to compute or learned first tend to be represented more strongly and densely than features that are more complex or learned later, even if all features are learned equally well. We also explore how these biases are affected by architectures, optimizers, and training regimes (e.g., in transformers, features decoded earlier in the output sequence also tend to be represented more strongly). Our results help to characterize the inductive biases of gradient-based representation learning. We then illustrate the downstream effects of these biases on various commonly-used methods for analyzing or intervening on representations. These results highlight a key challenge for interpretability $-$ or for comparing the representations of models and brains $-$ disentangling extraneous biases from the computationally important aspects of a system's internal representations.

1 Introduction

Representations are widely used to understand or improve computation, but their relationship to computation is nontrivial. The paper studies how feature properties and learning conditions bias representations despite features having equivalent computational roles.

1 Introduction

  • Features with equivalent computational roles can be represented differently depending on complexity, learning order, prevalence, and output position.The study trains models to predict multiple features through separate outputs while manipulating these properties.Source: §1 · pp. 2–3
  • Simpler-to-compute features tend to be represented more strongly than complex features, even when they play equivalent roles in model outputs.Source: §1 · p. 2
  • Features learned earlier tend to be represented more strongly, making learned representations path-dependent even when train and test behavior is similar.Source: §1 · p. 3
  • Bias patterns recur across models and optimizers, but interactions with architecture, optimizer, and task can shift or reverse them.Source: §1 · p. 3
  • Representation biases can affect interpretation, comparisons between systems, and attempts to alter representations, because conclusions may reflect extraneous learning details.Source: §1 · p. 3

2 Definitions & methods

The paper defines representations and features operationally to compare their computational roles with how they are encoded. Its methods manipulate feature and dataset properties, then measure how much of model representations each feature explains.

2 Definitions & methods

  • A representation is distributed activity across model units; a feature representation is the portion predictable from a particular feature, without requiring causal involvement in behavior.The authors later verify causal roles in some cases and use feature-explained variance as a measure of representational strength.Source: §2 · pp. 3–4

2.1 Methods

  • Models are trained to predict statistically independent input features through separate outputs or Transformer output sequences, with held-out accuracy used to check generalization.Source: §2.1 · p. 4
  • Training datasets systematically vary feature complexity, prevalence, or output position, with IID validation and test sets constructed for analysis and generalization checks.Source: §2.1 · p. 4
  • Feature-explained variance is measured by fitting linear regressions from feature values to representation activations on validation data and testing on held-out data.The method uses R2; statistically independent features avoid overcounting shared variance in practice.Source: §2.1 · p. 4
  • Both easy and hard features reach perfect held-out accuracy, yet the easy feature explains substantially more penultimate-layer variance.The easy feature is a single input; the hard feature is the sum of four inputs modulo 2.Source: Figure 2 · p. 5

3 MLPs on binary features

The MLP experiments test how feature complexity, learning order, prevalence, and training choices shape representations when models compute multiple features. Easier, earlier, and more prevalent features can dominate representations even when model behavior gives little indication of that imbalance.

3.1 An initial look at complexity bias

  • The easy linear feature dominates penultimate representations over the harder sum-mod-2 feature, even as the hard feature is learned and generalizes.The top two principal components cluster by the easy feature, while the hard feature is encoded in higher components.Source: §3.1 · p. 6, Figure 3 · p. 6

3.2 Is the effect due to feature complexity or simply learning order?

  • Pretraining the hard feature closes the representational gap somewhat, but the easy feature still dominates; pretraining the easy feature changes little.The easy feature is learned first under simultaneous training, so easy-feature pretraining largely preserves that order.Source: §3.2 · p. 7

3.3 Varying features and accounting for many ways they can be represented

  • Accounting for every hard-input pattern leaves the gap under simultaneous training, but hard-feature pretraining closes it, implicating both learning order and representational alternatives.The input-pattern analysis measures all possible patterns relevant to the hard feature, not just the hard function itself.Source: §3.3 · p. 7

3.4 Additional evaluations of complexity biases on unit level, and with additional features

  • Similar qualitative biases appear in per-unit analyses, more complex features yield sparser representations, and graded biases persist with more than two features.These additional evaluations are reported in the supplement.Source: §3.4 · p. 7

3.5 Feature prevalence also biases representations

  • Less prevalent features explain less representational variance than equally difficult common features, despite perfect accuracy and low loss across all features.Rare and common versions of each feature generalize equally well in the reported experiment.Source: §3.5 · pp. 7, 9

3.6 The effects of architecture and hyperparameters—generally weak but complex interactions

  • Deeper MLPs tend to widen easy–hard representation gaps, while width has little effect and the overall pattern is robust to tanh or rectifier nonlinearities.The architecture and hyperparameter effects are generally weak but can interact in complex ways.Source: §3.6 · p. 9
  • At convergence, dropout interacts with optimizer choice: SGD and AdaGrad retain easy-feature gaps, while Adam can show a slight hard-feature bias at moderate dropout.Earlier in training, easy features remain dominant even after hard features are learned and generalize well.Source: §3.6 · p. 9

4 Transformers on sequence tasks

The transformer experiments examine feature representations in sequence tasks, from character classifications to structured and grammar-generated sentences. Biases vary across encoder and decoder components, and output position and the number of possible feature values also matter.

4 Transformers on sequence tasks

  • In character-sequence tasks, encoder representations reflect both feature complexity and token count, while decoder representations favor simpler features and features reported earlier in the output.Output order has little effect in the encoder; sequential training also produces distinct shifts in encoder and decoder biases over training.Source: §4 · p. 10, Figure 8 · p. 10

4.1 Dyck languages: structural and token-level properties dominate in different model components

  • For Dyck-(20,10) tasks, decoder representations favor the simplest feature, but encoder representations explain most variance through maximum nesting depth.The plotted analysis controls for sequence length; feature variance across seeds does not appear strongly correlated with test accuracy.Source: §4.1 · p. 11, Figure 9 · p. 11

4.2 More language-like datasets reveal an additional output-range bias

  • In grammar-generated English sentences, encoders emphasize sentence structure, while decoders represent noun identities and valence more strongly than lower-range properties such as number or tense.The number of possible output values may strongly affect variance near the output, and the pattern is also visible in the encoder to a lesser extent.Source: Figure 10 · p. 11, §4.2 · p. 12

5 Vision

The vision experiments compare CNNs and ResNets on object properties and relations. Representation biases depend on architecture and task: color dominates in non-residual CNNs, whereas ResNets favor shape in one setting and both model types show weaker, different biases in relational tasks.

5 Vision

  • Non-residual CNNs favor color representations, whereas ResNets favor shape despite color being easiest and shape slowest to learn.Both biases are weaker at the penultimate classifier layer than at the final pooling layer.Source: §5 · pp. 12–13, Figure 11 · p. 12
  • Residual connections may shift biases toward features integrating information over larger areas, but the authors suggest these effects interact with input and output structure.The proposed account is consistent with stronger integration-related biases in transformer encoders, but not with shifts in residually connected MLPs.Source: §5 · p. 13

5.1 Relational vision tasks show similar complexity biases

  • For relational-image tasks, both architectures favor basic over relational features and color or texture over shape; these biases weaken by the penultimate layer.Biases are weaker with fewer output classes, and binary labels and added objects may contribute to reversing ResNet basic-feature biases.Source: §5.1 · pp. 13–14, Figure 12 · p. 13

6 Why should we care? Evaluating downstream impacts

The section tests how representational biases affect common ways of interpreting, comparing, simplifying, and reusing learned representations. Across these uses, features represented more strongly can shape analyses and downstream behavior even when both features are learned successfully.

6.1 The representations identified are causally sufficient and specific

  • Feature-specific interventions flipped each feature’s labels across test items without changing the other feature’s labels, supporting causal sufficiency and specificity.Smaller interventions sufficed to flip hard-feature labels, consistent with those features occupying less representational variance.Source: §6.1 · p. 14, Figure 13 · p. 15

6.2 Impact on model simplification for interpretability

  • With only two principal components, the model identified the easy feature perfectly while hard-feature accuracy stayed at chance until at least four components were retained.Low-variance features can be missed by representational simplifications, potentially making them less faithful to the original model’s computations.Source: §6.2 · pp. 14–15, Figure 14 · p. 15

6.2.1 Impact on feature visualization

  • Low-dimensional visualizations can favor simpler or earlier-learned features; adding easy features pushes the hard feature into higher PCs and can obscure it in t-SNE plots.With one easy feature, t-SNE performs somewhat better, but enough easy features make the hard feature difficult to identify.Source: §6.2.1 · p. 15

6.3 Impact on Representational Similarity Analysis

  • RSA similarity can track feature-representation biases more than shared computations: models computing the same SUM%2 tasks appeared less similar than models computing simple linear features.This result used Euclidean distance; cosine distance produced qualitatively similar results.Source: §6.3 · p. 16, Figure 15 · p. 16

6.4 Impact on downstream use of representations

  • As both features became more predictive, downstream classifiers increasingly preferred the easier feature, despite equal predictive strength in the training setup.At low predictivity, classifiers relied equally on both features; the pattern was similar for linear and nonlinear classifiers.Source: §6.4 · p. 17, Figure 16 · p. 17

7 Related work

Prior work has documented biases toward simpler, common, or shortcut features, often where features compete within one task. This work studies representational biases when independently useful features are computed, alongside related challenges in neuroscience, interpretability, and controlled algorithmic settings.

Implicit inductive biases towards simplicity

  • Earlier work on implicit simplicity bias links simpler functions to easier learning and suggests more complex features may have sparser representations (Valle-Perez et al., 2018).The cited account also proposes that randomly parameterized networks are biased toward simpler functions.Source: §7 · p. 17

Contribution relative to prior work

  • Unlike most prior work focused on redundant features competing for one task, this study examines independent computational roles and finds representational biases even when complex features are learned equally well.The passage also notes that behavioral and representational biases can follow distinct learning trajectories, including a ResNet shape-bias case.Source: §7 · pp. 17–18

Representational biases towards simple, common features

  • Prior studies report representational biases toward simple or prevalent features, while this work additionally examines output-position biases in transformers.Examples include feature prevalence effects and biases in contrastive learning (Xue et al., 2023) and boolean feature learning (Qiu et al., 2024).Source: §7 · p. 18

Shortcut learning

  • Shortcut-learning studies describe reliance on spuriously correlated features, often simpler than the intended solution, but usually examine features serving one task rather than independent computations.Examples include texture over shape in ImageNet models (Geirhos et al., 2018) and semantic heuristics over complex syntax (McCoy et al., 2019; 2020).Source: §7 · p. 18

Related implications for disentanglement and interpretability

  • Disentangling objectives may shift feature-level biases, while interpretability methods that treat representational variance as importance may inherit similar biases.Prior interpretability work has noted that activation simplifications such as PCA can yield less faithful interpretations on challenging test distributions (Friedman et al., 2023a).Source: §7 · p. 18

Analyzing representations on algorithmic tasks

  • Controlled algorithmic-task studies often confound feature complexity with utility; this work complements them across more tasks and architectures, with less circuitry analysis per problem.A concurrent ResNet study on ImageNet found broadly compatible patterns but offered less systematic control over task and input statistics (Fel et al., 2024).Source: §7 · pp. 18–19

Neuroscience

  • Neuroscience faces related representation-to-computation challenges, including salient movement signals and orientation biases that can affect downstream behavior.Prior work also reports aliasing between systems with different functions in RSA and representational divergence across architectures computing similar functions (Maheswaranathan et al., 2019).Source: §7 · p. 19

8 Discussion

The discussion argues that representational biases can shape interpretability and cross-system comparisons, while causal interventions do not eliminate all uncertainty. It also identifies limits to extending the findings beyond the settings studied.

8 Discussion

  • The discussion suggests that models with different representational biases may have dissimilar representations despite computing the same functions, complicating model–brain comparisons.The authors caution that brain representations may follow different learning rules and be shaped by other factors, such as energy efficiency.Source: §8 · p. 20
  • Representational biases may shape interpretability and neuroscience comparisons, and can lead analyses to over- or underestimate systems’ computational similarity.High-variance features receive priority in many interpretability methods; the authors also suggest that sparse-autoencoder reconstruction objectives favor high-variance features, potentially neglecting computation-relevant ones.Source: §8 · pp. 19–20
  • Representational biases can shift which features models use for downstream classification, showing that representation structure can influence the functions they learn.The discussion distinguishes computational, algorithmic, and implementational levels while noting that they interact.Source: §8 · p. 20
  • Causal interventions strengthen claims about feature roles but do not fully resolve representational uncertainty, especially for nonlinear features with many possible encodings.Intervention hypotheses often originate in correlational analyses or task knowledge, and strong interventions on simpler features can preempt subtler features.Source: §8 · pp. 20–21

8.1 Limitations & future directions

  • The experiments use simple synthetic datasets and classification tasks, so biases may differ with realistic, entangled features or other objectives.The authors also identify limited contextuality as a difference from language modeling and call for study of richer contextual computation.Source: §8.1 · pp. 21–22
  • The models are toy-scale; larger models could show qualitatively different biases, while deeper models in the studied regimes showed somewhat larger biases.Hyperparameter choices shifted biases, but the authors did not observe those interactions in Transformers trained on more complex datasets.Source: §8.1 · p. 22
  • Linear analyses cannot detect information unavailable linearly from the augmented feature space, though the authors report that they explain most system variance and can support targeted behavioral changes.They identify nonlinear representations as an important direction for future work.Source: §8.1 · p. 22

9 Conclusions

The paper finds that feature properties and training choices bias learned representations, even after models learn the features. These biases may complicate interpretation and comparisons across systems.

9 Conclusions

  • Representations remain biased toward or away from features after models have learned them, with complexity, learning order, prevalence, and output position among the relevant properties.The biases may relate to implicit inductive biases in deep learning.Source: §9 · p. 22
  • The paper identifies representational biases as a potential challenge for interpreting learned representations and comparing them across systems.The stated scope includes machine learning, cognitive science, and neuroscience.Source: §9 · p. 22
  • The conclusion frames these biases as relevant to understanding gradient-based representation learning.Source: §9 · p. 22

A Supplemental methods

The supplement documents model and dataset construction, then adds MLP analyses showing that learning curves, representation variance, and feature-wise losses help characterize the reported biases.

A Supplemental methods

  • The supplemental materials identify the software used for training and analysis and provide a demonstration of the feature-complexity result.Models were trained in JAX with Flax; representation regressions used scikit-learn, and a demonstration notebook is linked.Source: A Supplemental methods · p. 30

A.1 Hyperparameters & architectural details

  • The main-experiment hyperparameters are tabulated, with additional architecture details for MLPs, transformers, CNNs, and ResNets.The supplement notes that some MLP hyperparameter choices are tested separately.Source: Table 1 · p. 30, §A.1 · p. 30

A.2 Datasets

  • Except where noted, datasets make features statistically independent and balance feature values within each dataset.Source: §A.2 · p. 31

A.2.1 Boolean

  • Boolean tasks use 64-bit input vectors and at least two boolean outputs, with nonoverlapping input units assigned to different features.Linear features are redundantly encoded across four units in the main experiments; the supplement reports this choice does not substantially affect results.Source: §A.2.1 · p. 31

A.2.2 Language

  • Language tasks include sequence-to-sequence feature classification, letter-string problems, grammatical sentences, and Dyck sequences.Targets include sentence properties and parsing-dependent features; the Dyck analyses control for sentence length because length correlates with some features.Source: §A.2.2 · pp. 31–32

A.2.3 Vision

  • Vision tasks use color, texture, and shape features, with stimuli for both single-object and relational tasks.For relational tasks, basic features are reported for the left object and relational features for objects on the right; objects are randomly offset within their allotted spaces.Source: §A.2.3 · pp. 32–33, Figure 17 · p. 33, Figure 18 · p. 33

B.1 MLP loss curves

B.2 Representation variance explained curves with and without normalization

  • Normalizing variance explained by current total variance preserves the qualitative biases while making early learning dynamics clearer.At initialization, the easy feature explains more variance than the hard feature, while all hard input patterns explain still more.Source: Figure 20 · p. 34, §B.2 · p. 34

B.3 MLP learning curves by training order

Learning order shapes representation variance even when learning curves indicate that features are being learned. Pretraining the hard feature changes its initial representation, but the easy feature soon overtakes it.

B.3 MLP learning curves by training order

  • When the hard feature is pretrained, it initially occupies more variance than the easy feature, which quickly overtakes it once learning begins.The learning-curve figure reports test accuracy and penultimate-layer variance explained across training orders.Source: §B.3 · p. 34, Figure 21 · p. 35
  • Considering all input patterns associated with the hard feature makes the pretraining effect clearer.Source: §B.3 · p. 34
  • The hard-feature pretraining curves also clarify the later-phase inflation observed for the easy feature in other training-order conditions.Source: §B.3 · p. 34

B.4 Per-unit representation variance analysis

Unit-level analyses show that easy-feature signals often remain prominent as harder features are learned, while harder features are represented more sparsely. With more features, complexity also shapes how features influence one another's variance.

B.4 Per-unit representation variance analysis

  • Units that ultimately carry the most difficult-feature variance often initially carry the most easy-feature variance, which is inflated as the difficult feature is learned.Only some initially easy-selective units later acquire hard-relevant signal; a few units initially selective for hard inputs become somewhat more hard-selective.Source: §B.4 · p. 35, Figure 22 · p. 36, Figure 23 · p. 37
  • Many units also amplify signal about other inputs or nonlinear patterns as the features are learned.The authors attribute this either to chance correlations in the dataset or to learning dynamics.Source: §B.4 · pp. 35–36

B.5 Feature complexity also biases sparsity of representations

  • Harder features have sparser representations by the study's coefficient-entropy measure, even when pretrained.The measure assigns higher sparsity when fewer units contribute to logistic-regression decoding.Source: §B.5 · p. 38, Figure 24 · p. 38

B.6 More features of varying complexity

  • With four features, adding simple features tends to suppress more complex features, while adding complex features tends to inflate simpler ones.Intermediate-difficulty features tend to inflate simpler features but interfere with more complex ones; the authors note that these effects cannot be fully disentangled in this experiment.Source: §B.6 · p. 38, Figure 25 · p. 39

B.7 The effects of MLP architecture and other hyperparameters

The MLP results are broadly similar across residual connections and tested input encodings, but depth, nonlinearity, and initialization scale can shift the representation gap.

B.7 The effects of MLP architecture and other hyperparameters

  • Deeper MLPs show somewhat larger easy–hard representation gaps, while width and aspect ratio have little apparent effect.Source: §B.7 · p. 39, Figure 26 · p. 40
  • Residual MLPs show qualitatively similar biases, whereas ResNets have strikingly different biases from CNNs.Source: §B.7 · p. 39, Figure 27 · p. 40
  • Tanh and Leaky ReLU yield qualitatively similar patterns, but tanh generally shows somewhat smaller gaps.Source: §B.7 · p. 39, Figure 28 · p. 41
  • Encoding the easy feature across four or eight input units slightly increases the gap, without changing the qualitative pattern.Source: §B.7 · p. 39, Figure 29 · p. 41
  • Smaller weight-initialization scales yield somewhat larger biases toward easy features.Source: §B.7 · p. 39

B.7.1 Dropout and optimizer interact for MLP architectures

Dropout’s effect on feature bias in MLPs depends on the optimizer and appears at convergence, while Transformer biases on sequence tasks remain more consistent with dropout. Sequentially trained Transformers show order-related representation biases and complex decoder dynamics.

B.7.1 Dropout and optimizer interact for MLP architectures

  • At convergence, dropout can reverse or erase MLP feature biases for some optimizers, while SGD and Adagrad retain easy-feature gaps; without dropout, optimizer patterns are qualitatively similar.The dropout interaction appears only at convergence; SGD and Adagrad fail to learn at high dropout rates.Source: §B.7.1 · pp. 41–42, Figure 31 · p. 42, Figure 32 · p. 43
  • Smaller weight initializations yield somewhat larger biases toward easy features.Source: Figure 30 · p. 42
  • Transformer feature biases on complex sequence tasks remain fairly consistent with dropout, unlike the optimizer-dependent MLP effects.The figure reports this comparison at convergence, with all models above 99% test accuracy.Source: §B.7.1 · p. 42, Figure 33 · p. 44

B.8 Transformer sequential learning curves

  • In sequentially trained Transformers, feature-order biases are visible but weaker than in MLPs at the penultimate layer, while the decoder shows interference, decay, and recovery.Decoder interference increases with each subsequent feature; weight sharing can raise a feature’s explained variance before its loss is enabled.Source: Figure 34 · p. 45, §B.8 · p. 45

B.9 Additional Dyck language results

Additional analyses find that Dyck-task variance explained does not clearly track test accuracy, and vision-task biases shift with input conditions. The analyses also show how easy features occupy leading components and can obscure harder features in visualizations.

B.9 Additional Dyck language results

  • For Dyck-language features, variance explained shows no clear correlation with test accuracy, although low variability in accuracy may limit detection.The analysis also reports representation variance without controlling for sequence length.Source: §B.9 · pp. 45–46, Figure 35 · p. 46, Figure 37 · p. 47

B.10.1 ResNet representational biases for shape are affected by the number of class labels and the presence of distractors

  • With binary labels alone, ResNets retain a weaker shape bias; adding distractor objects shifts biases toward color and texture, with distractors contributing more to the shift.In the learning curves, color first dominates at CNN spatial pooling, while ResNet color dominance is brief before shape overtakes it.Source: §B.10.1 · p. 47, Figure 38 · p. 48, Figure 39 · p. 49

B.11 Keeping all but the top-k principal components

  • Dropping top principal components harms easy-feature predictions at a smaller k than hard-feature predictions.The hard feature is more robust because the top components are occupied by the easy feature.Source: §B.11 · p. 49, Figure 40 · p. 49

B.12 Low-dimensional visualizations

  • As easy features are added, they occupy successive principal components and push the hard feature higher; t-SNE makes the hard feature easier to see than leading PCs, but it becomes difficult to discern with four easy features.In more realistic settings with many features, complex but important features may be swamped in typical visualizations.Source: §B.12 · p. 50, Figure 41 · p. 51, Figure 42 · p. 52

B.13 RSA result resilience

Related papers

Loading 2405.05847v3…