Source-linked AI summary
Enriching ImageNet with Human Similarity Judgments and Psychological Embeddings
Brett D. Roads, Bradley C. Love
TL;DR
Task-specific classification metrics provide limited evidence about how models’ internal representations align with human-perceived similarity. The paper assembles human similarity judgments and infers psychological embeddings using variational inference, ensembles, and active learning. Among evaluated supervised models, newer models’ classification improvements were not accompanied by increased correspondence with human similarity judgments.
Problem
Task-specific metrics such as classification accuracy provide limited evidence about how models’ internal representations align with human-perceived similarity.
Source: §1 · p. 1Method
Results
Among evaluated supervised models, newer models’ classification improvements were not accompanied by increased correspondence with human similarity judgments.
Source: §7 · p. 8Takeaways & Limitations
Human-similarity metrics can evaluate internal representations of models trained with either supervised or unsupervised procedures.
Source: §7 · p. 8Perfect triplet ranking is not expected because participants may disagree about similarity and individuals may be metrically inconsistent.
Source: §6.1 · p. 7
Abstract
from arXiv · showhide
Advances in object recognition flourished in part because of the availability of high-quality datasets and associated benchmarks. However, these benchmarks---such as ILSVRC---are relatively task-specific, focusing predominately on predicting class labels. We introduce a publicly-available dataset that embodies the task-general capabilities of human perception and reasoning. The Human Similarity Judgments extension to ImageNet (ImageNet-HSJ) is composed of human similarity judgments that supplement the ILSVRC validation set. The new dataset supports a range of task and performance metrics, including the evaluation of unsupervised learning algorithms. We demonstrate two methods of assessment: using the similarity judgments directly and using a psychological embedding trained on the similarity judgments. This embedding space contains an order of magnitude more points (i.e., images) than previous efforts based on human judgments. Scaling to the full 50,000 image set was made possible through a selective sampling process that used variational Bayesian inference and model ensembles to sample aspects of the embedding space that were most uncertain. This methodological innovation not only enables scaling, but should also improve the quality of solutions by focusing sampling where it is needed. To demonstrate the utility of ImageNet-HSJ, we used the similarity ratings and the embedding space to evaluate how well several popular models conform to human similarity judgments. One finding is that more complex models that perform better on task-specific benchmarks do not better conform to human semantic judgments. In addition to the human similarity judgments, pre-trained psychological embeddings and code for inferring variational embeddings are made publicly available. Collectively, ImageNet-HSJ assets support the appraisal of internal representations and the development of more human-like models.
1. Introduction
ImageNet-HSJ provides human similarity judgments to assess models’ internal representations beyond task-specific outputs. Psychological embeddings support active learning, data-sufficiency assessment, and model evaluation, with variational inference and ensembles addressing scale.
1. Introduction
- ImageNet-HSJ supplements ILSVRC [39] with informative human similarity judgments and supports assessment of arbitrary models.The dataset and companion pre-trained psychological embeddings are publicly available.Source: §1 · pp. 1–2
- Human-perceived similarity adapts to context, but dataset shortages and collection challenges have limited its use in computer vision development.Similarity can reflect related roles, shared perceptual properties, or interactions between objects.Source: §1 · p. 1
- Human–model similarity alignment offers a complementary task-general evaluation of internal representations across supervised, unsupervised, and self-supervised training.The proposed comparison focuses on representations preceding task-specific outputs rather than replacing metrics such as classification accuracy.Source: §1 · pp. 1–2
- Psychological embeddings model similarity judgments to guide active learning, assess data sufficiency, and evaluate models’ internal representations.Extending previous work [50], these embeddings combine stimulus representations with functions linking them to observed behavior.Source: §1 · p. 2
- Variational inference and ensembles extend existing embedding approaches to handle large numbers of stimuli.An open-source Python package is available for inferring variational psychological embeddings.Source: §1 · p. 2
2. Related work
ImageNet-HSJ extends human-similarity embedding research to larger stimulus sets and provides a multipurpose resource for comparing models with human judgments. Its variational-inference approach reduces computational costs and addresses rotation-related uncertainty artifacts in posterior sampling.
2. Related work
- An order-of-magnitude increase in stimulus-set size over prior THINGS work brings human-similarity embeddings to a dataset widely used in machine learning.THINGS research collected judgments for 1,854 unique images. Earlier embedding methods include kernel variants [50] and non-metric approaches.Source: §2 · p. 2
- ImageNet-pretrained VGG16 predicts human similarity ratings [31] and rankings above chance.Attarian et al., Peterson et al., and Sanders et al. further demonstrated that neural networks can be trained specifically to predict human similarity judgments with relatively high fidelity.Source: §2 · p. 2
- The authors aim to complement targeted comparisons between artificial neural networks and human behavior by providing a dataset intended to serve multiple purposes, not solely as an evaluation metric.Prior work compared classification performance [32], revealed differences in image-level confusion statistics and shape bias, and examined prediction of human-perceived typicality ratings.Source: §2 · p. 2
- Unlike Roads et al., this work samples an approximate variational posterior, reducing computational costs and avoiding rotation-related uncertainty artifacts associated with MCMC.These computational savings support larger stimulus sets. Rotation-invariant embeddings can rotate during MCMC sampling, artificially inflating uncertainty for peripheral points.Source: §2 · p. 2
3. ImageNet-HSJ dataset
ImageNet-HSJ supplements the ILSVRC 2012 validation set with quality-controlled ordinal human similarity judgments. Collection expanded from a class-representative seed subset to all 50,000 validation images, with versioned releases tracking coverage.
3. ImageNet-HSJ dataset
- ImageNet-HSJ adds quality-controlled ordinal human similarity judgments to the ILSVRC 2012 validation set, prioritizing evaluation as a prerequisite for model selection.Judgments were collected from a wide range of participants.Source: §3 · p. 2
3.1. Human similarity judgment task
- Participants ranked the two references most similar to a central query image among eight surrounding references.The 8-rank-2 format balances throughput and noise while remaining easy for most participants. Web-based sessions recruited through Amazon Mechanical Turk comprised 50 trials, targeted 10 minutes, and paid approximately 8.00 USD per hour.Source: §3.1 · pp. 2–3
3.2. Stimulus set
- 50,000 validation images received judgments after active learning initialized and converged on a 1,000-image seed subset containing one representative per class.Representatives maximized a pre-trained VGG19 network’s probability of correct classification within each class, with ties resolved randomly. Images on the ILSVRC 2014 exclusion list were ineligible for the seed subset.Source: §3.2 · p. 3
3.3. Versioning
- Version 0.2 expands seed-only version 0.1 to full-stimulus coverage, while version 1.0 is planned after full-dataset convergence.Version 0.2 includes all judgments collected at submission.Source: §3.3 · p. 3
4. Dataset collection
Dataset collection alternates information-guided trial selection, quality-controlled human judgments, and variational inference of psychological-embedding ensembles. Sampling heuristics make trial selection tractable, while confirmation trials help correct misplaced stimuli.
4. Dataset collection
- Active learning extends existing paradigms [50] through variational Bayesian inference, cycling between information-guided trial selection, human judgments, and embedding-ensemble updates.Initial trials are selected randomly under the embedding prior. Subsequent selection prioritizes informative trials rather than highly dissimilar image combinations likely to elicit inconsistent choices.Source: §4 · p. 3, Figure 2 · p. 4
4.1. Judgment collection
- 50-trial sessions used four mirrored-query catch trials to weight judgments and screen participants.Selecting the mirrored query first, second, or neither scored 1, .5, or 0; the mean catch score weighted every session trial. Sessions scoring < .5 and trials lasting less than 1 s were dropped; only premium participants (> .875) remained eligible, and scheduling sought one premium rating per session.Source: §4.1 · pp. 3–4
4.2. Embedding inference
- Variational free-energy minimization yields approximate embedding posteriors describing both epistemic and aleatoric uncertainty.Diagonal covariance matrices limit complexity. Sample weights linearly scale each trial’s log-likelihood contribution under a Luce’s ratio of strengths formulation.Source: §4.2 · p. 4
- Three equally weighted embeddings, each withholding a different 5% of training data, reduce dependence on initialization-sensitive solutions.The withheld data selected dimensionality by categorical cross entropy when reevaluated; otherwise, dimensionality carried forward. Each iteration reused the two best models’ weights and reinitialized the worst to reduce local-optimum risk.Source: §4.2 · p. 5
4.3. Trial selection
- Expected information gain measures the reduction in embedding Shannon entropy expected after observing a candidate trial, averaged across ensemble members.Posterior samples approximate the required integrals; heuristics reduce the otherwise infeasible search over all possible trials.Source: §4.3 · p. 5
- 828 query stimuli per iteration were sampled without replacement using frequency-adjusted entropy; references were sampled in proportion to ensemble-average expected similarity.For each query, 10,000 candidate trials drew from the 500 nearest neighbors (1% of the dataset); the top three by ensemble-average information gain were retained (2484 trials). Neighborhood restriction probes fine-grained similarity, while frequency adjustment discourages repeatedly sampling stimuli with high aleatoric uncertainty.Source: §4.3 · p. 5
- 828 additional confirmation trials per iteration counteracted the risk that neighborhood-restricted selection would leave stimuli embedded among dissimilar neighbors.Queries were sampled in proportion to ensemble-average entropy, with two references drawn from the 500 nearest neighbors and six from outside. Participant choices could reveal inappropriate neighborhoods and help relocate stimuli.Source: §4.3 · pp. 5–6
5. Evaluating dataset convergence
Convergence is assessed through coarse-grained loss and agreement within and between successive embedding ensembles. All three measures indicate convergence for the seed subset, but not for the full dataset.
5.1. Coarse-grained loss
- Coarse-grained loss measures categorical cross-entropy on randomly sampled similarity judgments, tracking generalization as additional data are collected.Random sampling rarely probes fine-grained similarity, so these observations are not treated as a conventional test set. Loss reaching a minimum plateau provides some evidence of convergence.Source: §5.1 · p. 6
5.2. Within-ensemble agreement
- Within-ensemble agreement examines learned similarity structure by averaging Pearson correlations between the three models’ expected pairwise similarity matrices.Correlations use upper-triangular matrix elements; comparable embeddings across models are expected once sufficient data have been collected.Source: §5.2 · p. 6
5.3. Consecutive-ensemble agreement
- Consecutive-ensemble agreement measures Pearson correlation between successive ensembles’ expected similarity matrices, testing whether additional data alter the inferred similarity structure.Once sufficient data have been collected, adding more should leave the inferred ensemble unchanged.Source: §5.3 · p. 6
5.4. Convergence results
- All measures indicate that the psychological embeddings for the seed subset have converged, whereas those for the full dataset have not converged (Figure 5.4).The seed subset and full dataset contain 1,000 and 50,000 unique stimuli, respectively; Figures 5.4a and 5.4b should not be directly compared because different coarse-grained sets are used. The dip in Figure 5.4b demonstrates embeddings’ susceptibility to local optima.Source: §5.4 · p. 6
6. Using the dataset to evaluate target models
The dataset evaluates model representations through triplet accuracy and correlation with a psychological embedding. ResNet50 performs best among the tested target models on both metrics, while psychological embeddings predict triplet judgments better than those models.
6.1. Triplet accuracy
- 13 triplet inequalities per 8-rank-2 trial support triplet accuracy, the proportion correctly ranked using a target model’s chosen distance function.This direct metric is less sensitive to dataset convergence and requires only a distance-function choice. Perfect accuracy is not expected because participants can disagree and individuals may lack metric consistency.Source: §6.1 · pp. 6–7, §6 · p. 6
6.2. Psychological embedding correlation
- Psychological embedding correlation uses Spearman correlation between upper-triangular pairwise similarity matrices to assess structure that coarse triplet accuracy misses, given sufficient observations.Researchers select an embedding and target-model similarity function; Spearman correlation avoids assuming linearly related similarities. Between psychological embeddings trained on identical data, expected similarity matrices have Spearman correlations of 0.81 for seed and 0.62 for full.Source: §6.2 · p. 7, §6.3 · p. 8
6.3. Target model results
- The demonstration evaluates supervised image classifiers alongside unsupervised DeepCluster (VGG16, fc2) [4].Except for DeepCluster, pretrained weights come from Keras Applications. Images receive model-specific preprocessing and resizing from the largest centered crop preserving aspect ratio; DeepCluster uses its repository’s weights and preprocessing.Source: §6.3 · p. 7
- ResNet50 leads the evaluated target models on triplet accuracy for both datasets and on psychological embedding correlation.All target models exceed chance triplet accuracy, with cosine distance outperforming L1 and L2 distance in every case. Cosine similarity produces the highest embedding correlation in most cases.Source: §6.3 · pp. 7–8, Table 2 · p. 8
- Psychological embeddings predict triplet judgments better than the target models, although their training on those judgments makes the comparison unequal.The embeddings are blind to the stimuli, whereas the other target models can leverage approximately one million images of experience.Source: §6.3 · p. 8, Table 2 · p. 8
7. Discussion
Human similarity judgments can evaluate supervised and unsupervised model representations, revealing that advances in ImageNet classification have not increased correspondence with human similarity judgments. Variational inference, ensembles, and active learning enabled psychological embedding inference at a larger scale.
7. Discussion
- An order of magnitude larger than previous work, the embedding-inference problem was made tractable through variational inference, ensemble models, and active learning.The method efficiently infers embedding spaces from human judgments for large numbers of items.Source: §7 · p. 8
- DeepCluster was among the worst-performing models, contrary to expectations that unsupervised representations would better match human judgments.The evaluation metrics apply to both supervised and unsupervised models, but only one unsupervised model was evaluated, limiting conclusions about unsupervised learning generally.Source: §7 · p. 8
- Newer supervised models improved ImageNet classification performance without increasing correspondence with human similarity judgments.Earlier classification improvements accompanied better correspondence with human behavioral and brain data. The authors note an analogous trend in Brain-Score rankings.Source: §7 · p. 8
- Human similarity judgments reveal context-dependent organization schemes, motivating potential applications of the embedding method and dataset beyond evaluating model representations.These additional applications are anticipated rather than demonstrated in this discussion.Source: §7 · p. 8