PaperlayerView exact revision on arXiv

Source-linked AI summary

Enriching ImageNet with Human Similarity Judgments and Psychological Embeddings

Brett D. Roads, Bradley C. Love

arXiv:2011.11015v1cs.CVcs.LG

TL;DR

Existing benchmarks emphasize task-specific outputs, leaving human-perceived similarity and internal representations less directly evaluated. The paper introduces ImageNet-HSJ with scalable psychological embeddings and finds that stronger classification performance does not reliably improve alignment with human similarity judgments.

  • Problem

    Task-specific metrics such as classification accuracy provide limited evaluation of whether diverse models’ internal representations align with human-perceived similarity.

  • Method

    The authors collect human similarity judgments and infer large-scale psychological embeddings using variational inference, ensemble models, and active learning.

  • Results

    Models trained with supervised or unsupervised procedures can be evaluated, but DeepCluster performed among the worst and newer classification gains did not increase correspondence with human similarity judgments.

  • Takeaways & Limitations

    ImageNet-HSJ provides a task-general basis for assessing model representations against human judgments across supervised and unsupervised learning procedures.

  • Takeaways & Limitations

    Human similarity judgments are not expected to rank all triplets consistently because participants may disagree and individuals may not be metric consistent.

Abstract

from arXiv · show

Advances in object recognition flourished in part because of the availability of high-quality datasets and associated benchmarks. However, these benchmarks---such as ILSVRC---are relatively task-specific, focusing predominately on predicting class labels. We introduce a publicly-available dataset that embodies the task-general capabilities of human perception and reasoning. The Human Similarity Judgments extension to ImageNet (ImageNet-HSJ) is composed of human similarity judgments that supplement the ILSVRC validation set. The new dataset supports a range of task and performance metrics, including the evaluation of unsupervised learning algorithms. We demonstrate two methods of assessment: using the similarity judgments directly and using a psychological embedding trained on the similarity judgments. This embedding space contains an order of magnitude more points (i.e., images) than previous efforts based on human judgments. Scaling to the full 50,000 image set was made possible through a selective sampling process that used variational Bayesian inference and model ensembles to sample aspects of the embedding space that were most uncertain. This methodological innovation not only enables scaling, but should also improve the quality of solutions by focusing sampling where it is needed. To demonstrate the utility of ImageNet-HSJ, we used the similarity ratings and the embedding space to evaluate how well several popular models conform to human similarity judgments. One finding is that more complex models that perform better on task-specific benchmarks do not better conform to human semantic judgments. In addition to the human similarity judgments, pre-trained psychological embeddings and code for inferring variational embeddings are made publicly available. Collectively, ImageNet-HSJ assets support the appraisal of internal representations and the development of more human-like models.

1. Introduction

This work introduces a dataset and psychological-embedding framework for evaluating arbitrary models against human-perceived similarity, complementing task-specific metrics with task-general assessments. The motivation is that human similarity judgments flexibly reflect context and rich world knowledge, making them relevant to more human-like machine representations.

  • Motivation: Human-perceived similarity flexibly adapts to context, reflecting a rich understanding of the world.For example, a beer bottle may be judged similar to cigarettes as age-restricted objects or to soda as a beverage.
  • Motivation: Task-general evaluation metrics can complement task-specific measures such as classification accuracy as models shift toward task-general representations.Internal representation metrics can support comparisons across supervised, unsupervised, and self-supervised training paradigms.
  • Motivation: More human-like machine thinking may improve human-machine interactions.
  • Contribution: The work aims to assemble a dataset embodying human-perceived similarity and demonstrate its use for assessing arbitrary models.It extends previous work [50] by using psychological embeddings to concisely model information contained in similarity judgments.
  • Contribution: Psychological embeddings include an embedding of the stimuli and functions linking that embedding to observed behavior.The paper identifies psychological embeddings as serving three different roles.

2. Related work

Prior work used human similarity judgments to infer embeddings, increasingly with larger and more naturalistic stimuli, while other studies compared artificial networks with human behavior. This work builds on active-learning methods for collecting similarity judgments by using variational inference to reduce computational costs and scale to larger stimulus sets.

  • Embedding human similarity judgments: Human similarity judgments have long been used to infer embeddings through kernel and non-metric algorithms, with recent studies increasing stimulus scale and naturalism.A recent THINGS study collected judgments for 1,854 unique images.
  • Comparing models with human behavior: Research comparing artificial neural networks with human behavior has examined classification performance, image-level confusions, shape bias, and human-perceived typicality ratings.
  • Efficient data collection: The approach builds on active-learning paradigms for human similarity judgments, replacing MCMC posterior sampling with direct sampling from a variationally inferred approximate posterior to reduce computational costs.The stated benefit is enabling scaling to larger stimulus sets.

3. ImageNet-HSJ dataset

ImageNet-HSJ augments the ILSVRC 2012 validation set with quality-controlled ordinal human similarity judgments for all 50,000 images. Its data collection uses an eight-reference, rank-two selection task, active-learning expansion from a 1,000-image seed subset, and staged versioning.

  • Dataset scope: ImageNet-HSJ is a versioned dataset of ordinal human similarity judgments supplementing the ILSVRC 2012 validation set, with quality-controlled judgments from many participants.The dataset focuses exclusively on the validation set because evaluation supports model-selection paradigms.
  • Data collection: Each collection session used 50 trials, lasted about 10 minutes, and compensated participants at approximately 8.00 USD per hour.Judgments were collected through a web application from participants recruited via Amazon Mechanical Turk.
  • Human task: Participants selected the two reference images most similar to a center query from eight surrounding images, ranking their first and second choices.The eight-reference, rank-two format was chosen for ease of use and a balance between throughput and noise.
  • Stimuli coverage: Judgments were collected for all 50,000 ILSVRC validation images by beginning with a 1,000-image seed subset and expanding after active-learning convergence.The seed contained one representative image per class, selected using VGG19 predictions.
  • Versioning: Release version 0.1 covers the seed subset, version 0.2 covers the full stimulus set with judgments available at submission, and version 1.0 is planned after convergence.This versioning scheme separates initial seed collection, full-dataset coverage, and the intended final release.

4. Dataset collection

ImageNet-HSJ collects human similarity judgments through active learning rather than random sampling, selecting trials for expected information gain and iteratively refining psychological embeddings. Quality control uses catch trials and session grading, while variational posteriors and embedding ensembles guide subsequent trial selection.

  • Active learning: Active learning iteratively selects trials, collects judgments, and infers psychological embeddings, targeting trials with maximum expected information gain rather than sampling randomly.Expected information gain is defined as the reduction in embedding entropy expected from adding a candidate trial; heuristics make search feasible despite the enormous trial space.
  • Quality control: Selected trials were partitioned into 72 sessions of 50 trials, each containing four strategically placed catch trials for quality assessment.Two catch trials appeared among the first 20 trials and two among the last 20 trials.
  • Quality control: Unsatisfactory sessions with average grades below .5 were discarded, while premium sessions above .875 kept participants eligible for additional work.The procedure targeted one premium rating for each session and excluded non-premium participants from further eligibility.
  • Embedding inference: Posterior uncertainty from variational inference identifies where the embedding is uncertain, and an equally weighted ensemble of three embeddings limits sensitivity to initialization and local optima.The posterior captures both epistemic and aleatoric uncertainty, while ensemble members provide more robust guidance for selecting future trials.

5. Evaluating dataset convergence

The study assesses convergence of psychological embeddings using coarse-grained loss, within-ensemble agreement, and across-iteration agreement. All measures indicate convergence for the seed subset but not for the full dataset, whose embeddings remain sensitive to local optima.

  • Measures: Convergence was assessed through coarse-grained loss, within-ensemble agreement, and across-iteration agreement as additional similarity data were collected.These measures track embedding changes, model agreement within an ensemble, and agreement across consecutive iterations.
  • Coarse-grained loss: Coarse-grained loss approaching a minimum across training iterations provides evidence that the psychological embeddings have converged.The coarse-grained trials are randomly sampled and contain few observations probing fine-grained similarity because highly similar image pairs are unlikely.
  • Embedding agreement: Within-ensemble agreement compares models’ implied pairwise similarity matrices, while across-iteration agreement uses Pearson correlations between expected similarity matrices from consecutive ensembles.Sufficient data should yield comparable embeddings within an ensemble and should prevent additional data from altering the inferred ensemble.
  • Results: All measures indicate that the seed-subset embeddings converged, whereas the full-dataset embeddings had not converged (Figure 5.4).The dip for the full dataset demonstrates susceptibility to local optima.

6. Using the dataset to evaluate target models

This section evaluates arbitrary model representations against ImageNet-HSJ using direct triplet accuracy and correlation with a psychological embedding. Across popular supervised models, ResNet50 performs best, while the psychological embedding achieves the highest triplet-rating accuracy despite being stimulus-blind.

  • Direct triplet evaluation: Triplet accuracy converts each 8-rank-2 judgment into implicit triplet inequalities and measures the proportion correctly ranked by a target model.For q : a > b, a triplet is correct when d(Zq, Za) < d(Zq, Zb), and mean correctness gives triplet accuracy.
  • Evaluation methods: Triplet accuracy is simple but coarse, because human judgments can disagree and individuals may not be metric consistent.A psychological-embedding comparison provides a stronger assessment when sufficient observations are available.
  • Evaluation results: ResNet50 has the highest triplet accuracy on both the seed and full datasets, with cosine distance outperforming L1 and L2 for every model.All evaluated models perform above chance.
  • Evaluation results: The psychological embedding performs best at predicting triplet ratings, despite being blind to the stimuli and trained on the triplet data.The comparison is not completely fair because other models leverage approximately one million images of experience; a hold-one-trial-out procedure would yield similar accuracy.
  • Evaluation results: ResNet50 is also the top performer under psychological-embedding correlation, and cosine similarity yields the best correlation for most target models.Correlations compare upper-triangular pairwise similarity matrices using Spearman correlation, avoiding an assumption of linear similarity relationships.

7. Discussion

The method efficiently infers large-scale psychological embedding spaces from human similarity judgments using variational inference, ensemble models, and active learning. These judgments evaluate supervised and unsupervised model representations, revealing that stronger ImageNet classification performance does not necessarily imply greater alignment with human similarity judgments.

  • Methodological contribution: The approach scales psychological embedding inference to a problem an order of magnitude larger than previous work through variational inference, ensemble models, and active learning.Assessing consistency between model representations and human judgments provides a useful evaluation metric for large numbers of items.
  • Model evaluation: DeepCluster was one of the worst-performing models despite the expectation that unsupervised representations would generalize better and better conform to human judgments.The metrics can evaluate representations learned through both supervised and unsupervised procedures, although only one unsupervised model was tested.
  • Model evaluation: Supervised models’ correspondence with human similarity judgments has not increased as newer models have improved ImageNet classification performance.Earlier classification models showed better correspondence with human behavioral and brain data, but this relationship did not continue for newer models.
  • Broader implications: Human similarity judgments reveal flexible, context-dependent ways humans structure the world and may support applications beyond evaluating model representations.The authors expect additional uses for both the inferred embedding spaces and the dataset.
Loading 2011.11015v1…