PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating (and improving) the correspondence between deep neural networks and human representations

Joshua C. Peterson, Joshua T. Abbott, Thomas L. Griffiths

arXiv:1706.02417v3cs.CV

TL;DR

Psychological theories have often been validated with simple artificial stimuli, and human representations are difficult to infer across the variability of natural images. The paper compares DNN representations with human similarity judgments and transforms the representations through convex optimization. DNNs capture some human representational structure but miss other structure, while transformed representations are used to predict natural-image category-learning difficulty.

  • Problem

    Psychological theory validation has often relied on simple artificial stimuli, and experiments can compare only a small subset of natural stimulus variability.

    Source: §1 · p. 3

  • Method

    The authors transform DNN representations to improve their fit to human similarity judgments using a convex optimization approach.

    Source: §3.1 · p. 10, §3.2 · p. 10

  • Results

    DNN representations capture some human representational structure but lack spatial and taxonomic structure found in human judgments.

    Source: §2.2 · p. 8

  • Takeaways & Limitations

    The resulting representations are used to predict the difficulty of learning natural-image categories.

    Source: §5 · p. 15

    The learned transformations are domain-contingent and transfer poorly to other domains, with correlations worse than those from the best untransformed representations.

    Source: §3.5 · p. 12

Abstract

from arXiv · show

Decades of psychological research have been aimed at modeling how people learn features and categories. The empirical validation of these theories is often based on artificial stimuli with simple representations. Recently, deep neural networks have reached or surpassed human accuracy on tasks such as identifying objects in natural images. These networks learn representations of real-world stimuli that can potentially be leveraged to capture psychological representations. We find that state-of-the-art object classification networks provide surprisingly accurate predictions of human similarity judgments for natural images, but fail to capture some of the structure represented by people. We show that a simple transformation that corrects these discrepancies can be obtained through convex optimization. We use the resulting representations to predict the difficulty of learning novel categories of natural images. Our results extend the scope of psychological experiments and computational modeling by enabling tractable use of large natural stimulus sets.

Keywords: Artificial Intelligence, Similarity, Categorization, Neural Networks

The title frames the paper as evaluating and improving how deep neural networks correspond to human representations.

Keywords: Artificial Intelligence, Similarity, Categorization, Neural Networks

1 Introduction

Psychological theories have often been validated using artificial stimuli, while natural images pose challenges for directly inferring human representations. This paper evaluates DNN representations against human similarity judgments, transforms them to improve correspondence, and uses the resulting representations to predict natural-image category learning difficulty.

1 Introduction

  • Psychological theory validation has largely relied on simple artificial stimuli, and experiments cannot compare more than a relatively small set of stimuli.This limits representation estimates across the variability of natural stimuli.Source: §1 · p. 3
  • Human psychological representations are not directly observable, and neural representations are not necessarily a gold standard for capturing their structure.The paper therefore evaluates DNN representations against human behavior, including similarity judgments.Source: §1 · p. 4
  • The paper evaluates DNN and human similarities for spatial and taxonomic structure, identifying both captured aspects and differences.It then considers whether transforming deep representations can better model human representations.Source: §1 · pp. 4–5
  • The resulting approach is used to predict how difficult it is for people to learn categories of natural images.The authors present this as a way to extend psychological theory validation to large sets of complex natural stimuli.Source: §1 · p. 5

2 Experiment 1: Evaluating the correspondence between representations

Experiment 1 compared human similarity judgments with DNN representations across six natural-image domains and examined their spatial and taxonomic structure. Classification-network representations provided a first approximation, but missed some structure found in human judgments; performance varied across networks and layers.

2 Experiment 1: Evaluating the correspondence between representations

  • Across six natural-image domains, the experiment evaluated DNN predictions of human similarity judgments.The stimuli covered animals, automobiles, fruits, furniture, vegetables, and a cross-cutting “various” set.Source: §2 · p. 5

2.1.1 Stimuli.; 2.1.2 Procedure.

  • Each domain contained 120 total images; each image pair in each category received exactly 10 unique ratings (7,140 pairs; 71,400 ratings per category), yielding six 120×120 similarity matrices after averaging judgments.Source: §2.1.1 · p. 5, §2.1.2 · p. 6

2.1.3 Deep neural network representations.

  • The study compared pretrained classification networks and other image representations using feature activations to approximate pairwise similarity.The feature sets included final hidden-layer activations and a 200-dimensional BiGAN latent representation.Source: §2.1.3 · pp. 6–7

2.2 Results and Discussion

  • Raw deep representations gave a reasonable first approximation of human similarity judgments, with VGG slightly better on average among classification networks.BiGAN corresponded less well than shallow HOG+SIFT features.Source: §2.2 · pp. 7–8
  • Human representations show distinguished spatial clusters and intuitive taxonomic structure that are absent from DNN representations.The comparison used spatial projections and dendrograms to reveal the differences.Source: §2.2 · p. 8
  • For all datasets, AlexNet’s final hidden layer was its best-performing layer, while performance fell near the classification outputs.The authors interpret this as evidence that relatively high-level, non-semantic information is most relevant to the human judgments studied.Source: §2.2 · pp. 8–9

3 Transforming deep representations

Raw deep-network representations predict human similarity judgments reasonably well, and feature-weight transformations increase their alignment with psychological representations. The section develops regularized diagonal weighting, reports improved within- and cross-domain predictions, and tests transfer across domains.

3 Transforming deep representations

  • Raw DNN representations already predict human similarity reasonably well, motivating their transformation to improve alignment with psychological representations.With nonnegative weights, square roots of the weights can rescale features directly, although the reported results do not impose that constraint.Source: §3 · p. 9, §3.1 · p. 10

3.1 Transforming representations

  • The method substitutes DNN features for F and learns feature weights w; diagonal W models each image-pair similarity as a weighted sum of feature products.The diagonal-weight formulation uses fewer parameters than a full W matrix and is chosen to reduce data and regularization needs.Source: §3.1 · pp. 9–10

3.2 Learning the transformations

  • L2 regularization makes fitting thousands of DNN feature weights a convex ridge-regression problem, with the penalty selected by six-fold cross-validation.Gradient descent optimizes squared reconstruction error plus a weighted w^T w penalty.Source: §3.2 · pp. 10–11

3.3 Improvements through feature adaptation

  • Transformed representations explained nearly twice the variance across datasets and largely reconstructed human taxonomic structure and spatial clustering.The reported variance gains followed cross-validated prediction; the VGG scores are reported in Table 1.Source: §3.3 · p. 11

3.4 Additional baseline models

  • Shuffled-design baselines never reached 0.01 variance explained, which the authors take as evidence that regularization controlled overfitting.The tests separately permuted rows, columns, or both while leaving target similarities unchanged.Source: §3.4 · pp. 11–12

3.5 Inter-domain transfer

  • 30% to 60%: learning a transformation across six domains doubled variance explained in human similarity judgments; leave-one-domain-out prediction showed similar improvements.The leave-one-out control shared no images or similar images between training and test sets.Source: §3.5 · p. 12

4 Experiment 2: Predicting the difficulty of learning categories of natural images

The experiment tests whether transformed DNN similarities predict how readily people learn categories of new natural images. Participants learned categories derived from transformed DNN spaces more readily than those derived from raw DNN spaces, while the transformation effect reversed for HOG+SIFT features.

4 Experiment 2: Predicting the difficulty of learning categories of natural images

  • A simple linear transformation brought DNN similarity predictions close to inter-rater reliability and corrected qualitative differences between raw DNN and psychological representations.The authors conclude that DNN features may support psychological experiments using natural images.Source: §4 · pp. 12–13

4.1.1 Stimuli.

  • New image sets were clustered by k-means from transformed, raw, or HOG+SIFT similarities, using k=2, 3, or 4 across six domains.The transformed representation was applied to new images, with similarities used as image representations for clustering.Source: §4 · p. 13, §4.1.1 · p. 13

4.1.2 Procedure.

  • Participants categorized images into predefined clusters and received correctness feedback after each response.They viewed a random image sequence and could take as long as they wanted to respond.Source: §4.1.2 · p. 14

4.2 Results and Discussion

  • F1,1404 = 66.28, p < .0001: categorization was higher for transformed than raw DNN spaces, while the transformation effect reversed for HOG+SIFT.The feature-set-by-transformation interaction was significant (F5,2845 = 65.22, p < .0001).Source: §4.2 · p. 14
  • F4,5616 = 752.91, p < .0001: performance changed across learning blocks, with an interaction likely reflecting faster first-block gains for transformed representations.The block-by-transformation interaction was F4,5616 = 5.96, p < .0001.Source: §4.2 · p. 14

5 General Discussion

The framework applies classic psychological methods to DNN representations: these networks approximate human similarity judgments but diverge systematically, while a simple reweighting can improve their predictions and extend them to new stimuli and category-learning behavior.

5 General Discussion

  • The framework combines DNN representations with psychological methods to study cognition using complex, naturalistic stimuli and may inform learning-software design.Source: §5 · pp. 15–16
  • DNN representations approximate human similarities but diverge systematically, including by lacking taxonomic representational information.Source: §5 · p. 15
  • A simple linear reweighting can make DNN representations predict human similarity judgments extremely well, without complex feature learning.The transformation is constrained to reweighting features and can be interpreted as dimensional attention.Source: §5 · p. 15
  • The resulting models transfer to new stimuli and predict complex behaviors, including the ease of category learning.Source: §5 · p. 15
  • Because natural-image domains differ from the DNNs’ 1,000-class training context, features may become more or less salient when people judge images within a domain.Source: §5 · p. 15

Related papers

Loading 1706.02417v3…