PaperlayerView exact revision on arXiv

Source-linked AI summary

What shapes feature representations? Exploring datasets, architectures, and training

Katherine L. Hermann, Andrew K. Lampinen

arXiv:2006.12433v2cs.LGstat.ML

TL;DR

The paper asks which features models represent and use when inputs contain useful and irrelevant information, a question relevant to decisions and adaptable representations. Using synthetic datasets with controlled feature–label relationships, it studies representations before and after training. The results show that models selectively enhance, suppress, and preferentially retain features according to both task relevance and ease of decoding.

  • Problem

    The paper asks which task-relevant and task-irrelevant features models represent and use, because this bears on model decisions and versatile representations.

  • Method

    The authors use synthetic datasets with controlled feature–label relationships and analyze model representations using decodability and representational similarity.

  • Results

    Models enhance target features, partially suppress non-target features, and preferentially represent redundantly predictive features that are more linearly decodable before training.

  • Takeaways & Limitations

    Untrained representations can help predict feature preferences, while training task and feature difficulty shape both what models represent and how consistently they represent it.

  • Takeaways & Limitations

    The interpretation that untrained-feature decodability reflects inductive bias and predicts preservation after training is stated as a hypothesis, and validation correlations follow the data distribution.

Abstract

from arXiv · show

In naturalistic learning problems, a model's input contains a wide range of features, some useful for the task at hand, and others not. Of the useful features, which ones does the model use? Of the task-irrelevant features, which ones does the model represent? Answers to these questions are important for understanding the basis of models' decisions, as well as for building models that learn versatile, adaptable representations useful beyond the original training task. We study these questions using synthetic datasets in which the task-relevance of input features can be controlled directly. We find that when two features redundantly predict the labels, the model preferentially represents one, and its preference reflects what was most linearly decodable from the untrained model. Over training, task-relevant features are enhanced, and task-irrelevant features are partially suppressed. Interestingly, in some cases, an easier, weakly predictive feature can suppress a more strongly predictive, but more difficult one. Additionally, models trained to recognize both easy and hard features learn representations most similar to models that use only the easy feature. Further, easy features lead to more consistent representations across model runs than do hard features. Finally, models have greater representational similarity to an untrained model than to models trained on a different task. Our results highlight the complex processes that determine which features a model represents.

1 Introduction

The paper asks how models select and represent task-relevant, irrelevant, unreliable, and redundantly predictive features, and how representations change with training. These questions matter for generalization, model decisions, fairness, and comparisons across models or neural data.

  • Untrained models can already contain useful feature information, raising questions about what training preserves, enhances, or suppresses.
  • Feature selection matters because shortcut features can solve training tasks while failing to generalize robustly.
  • Understanding feature representations is relevant to model computation, model comparison, and comparison with neural data.
  • Synthetic datasets let the authors directly control relationships between multiple input features and task labels.
  • The paper reports that training enhances task-relevant features while suppressing task-irrelevant and some task-relevant features.

2 Related Work

Prior work shows that models may compress, preserve, or preferentially learn different features, while untrained networks already encode substantial task-relevant structure. This paper connects those observations to feature preferences during training.

  • Prior theory proposes that models may compress task-irrelevant information, but empirical work has also decoded category-orthogonal features from trained CNNs.
  • Theoretical studies suggest models learn higher-variance or higher-level features before lower-variance or lower-level features.
  • Deep networks may favor simple functions, potentially reflecting inductive biases associated with optimization and overparameterization.
  • Shortcut-learning research shows that models can prefer texture over shape, unlike people, although data augmentation can reduce this bias.
  • Untrained networks already support transfer, classification, denoising, inpainting, and explanations of neural data, motivating analysis of how training changes their representations.

3 Does feature selection happen by enhancement or suppression?

The experiments test whether training changes feature sensitivity by enhancing target features, suppressing non-target features, or both. Across synthetic visual datasets and architectures, target features become more decodable while non-target features are only partially suppressed.

  • The experiments compare feature decodability before and after training models to classify shape, texture, or color.
  • The synthetic datasets contain images with multiple controllable visual features, including shape, texture, and color.
  • Linear decoders map frozen layer activations to feature labels, measuring which visual features are represented in each model layer.
  • Across datasets and architectures, visual features were decodable above chance from upper layers of untrained models; color was most decodable, followed by shape and texture.
  • In the Navon dataset, shape and texture exceeded 60% decoding accuracy from AlexNet’s final convolutional layer, versus 4.3% chance.
  • Training generally enhanced target-feature decodability and partially suppressed non-target features, which remained above chance.

4 What if multiple features are predictive?

When multiple features predict the label, models preferentially represent features that are more decodable or easier to learn, sometimes suppressing less preferred features. This preference can override predictivity, while irrelevant inputs remain partially represented.

  • 4.3 Do models prefer reliable but difficult features, or easy but less reliable ones?: Feature difficulty was operationalized by network complexity: the easy feature was linearly extractable, whereas the difficult feature required an XOR computation.The easy feature's predictivity varied from 0.5 to 1, while the difficult feature's predictivity was fixed at 0.9.
  • 4.1 What if two features redundantly predict the label?: Models trained on perfectly predictive feature pairs preferentially represent one feature, with color preferred over shape and shape over texture.These preferences align with the rank-order decodability of the features in untrained models.
  • 4.2 What if one feature perfectly predicts the label, but another only partially predicts it?: Target features are enhanced during training, whereas correlated non-target features are generally preserved or suppressed.Suppression remains approximately constant across correlation strengths until very high correlations; at conditional probability 0.9, correlated non-target features were slightly enhanced.
  • 4.3 Do models prefer reliable but difficult features, or easy but less reliable ones?: An easier feature can suppress a more reliable but difficult feature before their predictivities are matched.The model preferred the easy feature at predictivity 0.8 even though the difficult feature had predictivity 0.9.
  • 4.3 Do models prefer reliable but difficult features, or easy but less reliable ones?: Even when a feature was not predictive, individual input units associated with it remained decodable at around 80% accuracy, while nonlinear combinations were difficult to recover.Thus, irrelevant inputs were not completely suppressed.

5 What affects the representational similarity between models?

Representational similarity depends more on the task than on architecture or layer, with same-task models more similar than different-task models. Similarity also varies with feature difficulty and predictivity.

  • 5 What affects the representational similarity between models?: RSA compares model layers by correlating the upper triangles of representational dissimilarity matrices built from stimulus-evoked activations.Each RDM entry uses correlation distance between activation patterns for a pair of stimuli.
  • 5 What affects the representational similarity between models?: Models trained on matching tasks have more similar representations than models trained on different tasks, and task influences similarity more than architecture or layer.Texture-task models were less similar to one another than shape-task models or even two untrained models.
  • 5 What affects the representational similarity between models?: Similarity between texture-task models was 0.408 [0.394, 0.420], compared with 0.624 [0.615, 0.632] for shape-task models and 0.619 [0.542, 0.702] for untrained models.Values are means with bootstrap 95%-CIs.
  • 5 What affects the representational similarity between models?: In binary tasks, models trained with a highly predictive easy feature were more similar across runs than models trained with an unpredictive easy feature.Multi-task models resembled models trained with the highly predictive easy feature, even while also representing the difficult feature.

6 Conclusion

The paper concludes that inductive biases and training jointly shape feature representations: models often favor easier-to-decode features, suppress unused features, and exhibit representational similarity dominated by easier features.

  • Multi-task models most closely resemble models trained with the predictive easy feature, despite also being trained to output a difficult feature.
  • When multiple features redundantly predict labels, models principally enhance the feature most decodable from the untrained model.
  • Models may favor an easier feature over a harder, more predictive feature, while unused features can be suppressed without becoming completely undecodable.
  • Practitioners should not assume that label-relevant features will be used or represented, and untrained representations may help predict feature use.
  • Representational similarity analyses require care for models trained on multiple features or tasks because similarity is dominated by easier features.

Broader Impact

Understanding which representations models use is important for evaluating decision correctness and safety, especially under distribution shift and potential reliance on socially discriminatory features.

  • Understanding model representations supports assessment of decision correctness and safety when deployment inputs differ from training inputs.
  • Predicting which features models use is relevant to addressing ethical concerns about reliance on socially discriminatory features.

A Supplemental Figures

Supplementary experiments across Navon, Trifeature, correlated-feature, and redundant-predictor settings show that training enhances target features while often suppressing non-target or redundant features.

  • A.1 Feature decodability: Target features are enhanced and non-target features are suppressed in ResNet-50 models trained on the Navon dataset.Decoding compares untrained, shape-trained, and texture-trained models; chance is 4.3%.
  • A.1 Feature decodability: Non-target features are suppressed in post-pool representations of ResNet50 models trained to classify shape, texture, or color on Trifeature.The figure reports decoding for all three features across the three classification tasks.
  • A.2 Correlated features: Correlated non-target features are strongly suppressed in ResNet-50 when shape and color or shape and texture predict the target together.Target features are enhanced while correlated non-target features are suppressed, matching the pattern observed with AlexNet.
  • A.3 Redundant predictors: ResNet-50 preferentially learns one feature when feature pairs redundantly predict labels, with color more decodable than shape and shape more decodable than texture.
  • A.3 Redundant predictors: AlexNet and ResNet-50 sometimes suppress a feature that perfectly predicts the label when it is redundant with another feature.AlexNet suppresses shape or texture in specified layers, while ResNet-50 suppresses shape in the post-pool layer.

A.1 Binary feature tasks

Binary-feature experiments examine how feature difficulty, predictivity, nonlinear decoding, and task composition shape learning dynamics and representational similarity.

  • Feature-learning dynamics: An easier feature with predictivity 0.65 is decodable above chance before training, while a harder feature with predictivity 0.9 is not.The easy feature is learned first, with early spikes in its test performance and decodability.
  • Feature recovery: Nonlinear decoders trained on 2048 examples still fail to recover a difficult feature when an easier feature suppresses it.Nonlinear decoding offers some advantage over linear decoding, especially at moderately high easy-feature predictivity.
  • Representational similarity: Representational similarity is most sensitive to the target feature even when another feature is relatively strongly correlated with it.The comparison uses matched architecture and layer with five runs per condition.
  • Representational similarity: Additional analyses show that intermediate easy-feature predictivity values interpolate between main-text cases while retaining a bias toward easier features.The supplementary analyses expand the main-text results using multiple approaches.
  • Representational similarity: Different hidden-unit solutions for XOR can produce different representational dissimilarity matrices, illustrating why nonlinear tasks may yield lower representational similarity.

B Methods

The study decodes representations from specified layers of trained models using feature datasets, with decoder training procedures tailored to dataset and analysis type.

  • Layer definitions: AlexNet representations were decoded from pool3, fc6, or fc7, while ResNet-50 representations were decoded before or after global average pooling.The AlexNet layers include the final convolutional output and the first two classifier layers; ResNet-50 uses pre-pool and post-pool activations.
  • Training procedure: Decoder inputs were activations from a layer of a trained, frozen model responding to dataset-normalized images, or unnormalized images for untrained models.Decoders were trained and validated on Trifeature or Navon datasets whose feature sets were uncorrelated.
  • Training procedure: 250-epoch decoders used Adam with batch size 64 across six learning rates and six weight decays, selecting the highest-validation-accuracy configuration.For trained models, selection used mean validation accuracy across five cross-validation splits.
  • Training procedure: Binary-feature decoders were trained for 5000 epochs at learning rate 10^-3 without weight decay or hyperparameter search, while nonlinear decoders used one 64-unit hidden layer for 20000 epochs.The nonlinear decoders used a leaky-rectifier nonlinearity.

B.2 Datasets

The datasets use controlled binary features and labels, with synthetic visual stimuli spanning combinations of colors, shapes, and textures; representational analyses use independent or broadly sampled inputs and multiple similarity measures.

  • Datasets: Trifeature stimuli span the combinations of colors, shapes, and textures used in the dataset.The appendix includes sample images illustrating this feature range.
  • Datasets: Inputs were 32-element binary vectors divided into two 16-input domains, with labels probabilistically related to one feature from each domain.Each feature had a defined predictivity, and labels were generated by flipping an initial binary label before sampling matching domain inputs.
  • Representational analyses: Trifeature representational similarity analyses used 10 examples per feature combination from 100,000 uncorrelated images, with possible overlap of no more than 10% with train and validation sets.The passage notes that completely excluding overlap was difficult because of the small training-set sizes.
  • Representational analyses: Binary-feature representational similarity analyses used a newly sampled dataset independent of model training and evaluation data, and its feature predictivity did not affect RSA.RSA used activation patterns rather than labels.
  • Representational analyses: CKA, Euclidean-distance RSA, and Spearman rank correlation produced similar results to the primary representational-similarity analysis.These comparisons are reported in Fig. A.13.
Loading 2006.12433v2…