PaperlayerView exact revision on arXiv

Source-linked AI summary

Partial success in closing the gap between human and machine vision

Robert Geirhos, Kantharaju Narayanappa, Benjamin Mitzkus, Tizian Thieringer, Matthias Bethge, Felix A. Wichmann, Wieland Brendel

arXiv:2106.07411v2cs.CVcs.AIcs.LGq-bio.NC

TL;DR

The paper asks whether machine vision is closing its robustness gap with human vision beyond standard benchmarks. Using human experiments across 17 OOD datasets and models varying in objectives, architectures, and training-data scale, it finds that the robustness gap is closing while image-level consistency differences remain.

  • Problem

    Machines surpassed humans on ImageNet but remained less robust on challenging out-of-distribution data, motivating tests of progress toward human-like vision.

  • Method

    The study compares human observers with models differing in objective function, architecture, and training dataset size across 17 OOD datasets, releasing the datasets and evaluation code as a benchmark.

  • Results

    The OOD distortion robustness gap is closing: the best models now match or exceed human accuracies, although an image-level consistency gap remains.

  • Takeaways & Limitations

    The results provide cautious optimism that behavioural differences between human and machine vision are narrowing, while the benchmark enables future progress tracking.

  • Takeaways & Limitations

    The study focuses on non-adversarial OOD robustness and limits human presentation times to 200 ms, so human ceiling performance might be higher with more time.

Abstract

from arXiv · show

A few years ago, the first CNN surpassed human performance on ImageNet. However, it soon became clear that machines lack robustness on more challenging test cases, a major obstacle towards deploying machines "in the wild" and towards obtaining better computational models of human visual perception. Here we ask: Are we making progress in closing the gap between human and machine vision? To answer this question, we tested human observers on a broad range of out-of-distribution (OOD) datasets, recording 85,120 psychophysical trials across 90 participants. We then investigated a range of promising machine learning developments that crucially deviate from standard supervised CNNs along three axes: objective function (self-supervised, adversarially trained, CLIP language-image training), architecture (e.g. vision transformers), and dataset size (ranging from 1M to 1B). Our findings are threefold. (1.) The longstanding distortion robustness gap between humans and CNNs is closing, with the best models now exceeding human feedforward performance on most of the investigated OOD datasets. (2.) There is still a substantial image-level consistency gap, meaning that humans make different errors than models. In contrast, most models systematically agree in their categorisation errors, even substantially different ones like contrastive self-supervised vs. standard supervised models. (3.) In many cases, human-to-model consistency improves when training dataset size is increased by one to three orders of magnitude. Our results give reason for cautious optimism: While there is still much room for improvement, the behavioural difference between human and machine vision is narrowing. In order to measure future progress, 17 OOD datasets with image-level human behavioural data and evaluation code are provided as a toolbox and benchmark at: https://github.com/bethgelab/model-vs-human/

1 Introduction

The paper asks whether advances in deep learning are narrowing the human–machine vision gap, arguing that IID accuracy is no longer sufficient and motivating broad OOD evaluation. It introduces a benchmark spanning newer objectives, architectures, and dataset scales, finding improved robustness but persistent image-level consistency differences.

  • Models increasingly match or exceed humans on IID benchmarks, making out-of-distribution evaluation necessary for assessing the human–machine vision gap.
  • The benchmark examines machine-learning developments across objective function, architecture, and training dataset size.
  • 17 OOD datasets with human comparison data and open-source evaluation code support future benchmarking of machine vision.
  • The distortion robustness gap is closing, with leading models matching or exceeding human feedforward performance on most investigated OOD datasets.
  • A substantial image-level consistency gap remains, although it narrows on some datasets as training dataset size increases.

2 Methods: datasets, psychophysical experiments, models, metrics, toolbox

The study combines controlled psychophysical experiments with a broad model benchmark and metrics that capture both aggregate accuracy and image-level agreement. It evaluates human observers and diverse models across OOD datasets, including measures designed to distinguish shared decisions from mere accuracy overlap.

  • Psychophysical experiments: 85,120 psychophysical trials across 90 observers provide human data for 17 generalisation datasets.
  • Psychophysical experiments: Human observers classified centrally presented images from 16 basic categories under controlled laboratory conditions.
  • Models: The benchmark tests 52 models spanning supervised, self-supervised, adversarially trained, semi-weakly supervised, transformer, and language-image approaches.
  • Metrics: OOD accuracy and accuracy difference compare aggregate performance between models and human observers.
  • Metrics: Observed consistency measures the fraction of images that humans and a model classify either both correctly or both incorrectly.
  • Metrics: Error consistency measures whether human–model agreement exceeds the above-chance overlap expected from independent decision makers with matched accuracy.
  • Benchmark & toolbox: The open-source modelvshuman toolbox supports benchmarking PyTorch and TensorFlow models against the human OOD dataset suite.

3 Robustness across models: the OOD distortion robustness gap between human and machine vision is closing

Across 17 OOD datasets, the distortion robustness gap between humans and machines is closing, especially for models trained on large-scale data. However, model behavior still differs from human perception in image-level errors, while architecture, training objectives, and dataset size produce distinct trade-offs.

  • The OOD distortion robustness gap between human and machine vision is closing, especially for models trained on large-scale datasets.The comparison spans humans and models differing in objective function, architecture, and training dataset size.
  • Adversarial training increases shape bias toward human-like behavior but makes models more vulnerable to large-scale image degradations.A 90-degree rotation produces a 50% classification-accuracy drop in the strongest adversarially trained models.
  • The best vision transformer, ViT-L trained on 14M images, exceeds human OOD accuracy averaged across 17 datasets.Vision transformers trained on 1M images also outperform standard convolutional models, with additional gains at 14M images.
  • Training on substantially more data narrows the gap to human OOD accuracies more than advanced architectures in standard models.BiT-M, Noisy Student, and SWSL used 14M, 300M, and 940M images, respectively, and exceeded a ViT-S trained on 1M images.
  • CLIP scores close to humans across all reported metrics, most strikingly in error consistency, combining 400M images, language-image supervision, and a vision transformer backbone.Noisy Student, trained on a comparably large dataset, performs well on OOD accuracy but poorly on error consistency.

4 Consistency between models: data-rich models narrow the substantial image-level consistency gap between human and machine vision

Models trained on larger datasets can narrow human–machine image-level error consistency gaps, but this progress depends strongly on the dataset and does not eliminate the gap across all OOD conditions.

  • Models trained on large-scale datasets, including CLIP and SWSL, make more human-like errors than standard models on some OOD datasets.On sketch images, data-rich models blur the boundary between human and model error patterns.
  • Most models cluster together in their categorisation errors despite differences in architecture and objective function.This includes standard supervised, self-supervised, adversarially trained, and vision-transformer models.
  • 12 datasets retain a large human–machine consistency gap even where model accuracies exceed human accuracies.Greater robustness does not improve error consistency on these datasets.
  • On five datasets, OOD accuracy is a near-perfect predictor of error consistency, and data-rich models such as CLIP, SWSL, and BiT narrow the gap.These datasets are sketch, silhouette, edge, cue conflict, and low-pass filtering.
  • Training dataset appears to have the strongest effect on model decision boundaries as measured by error consistency, although the underlying reason remains unresolved.Shared shortcut opportunities may cause models trained on similar data to converge to similar solutions.

5 Discussion

The study reports partial progress toward closing the human–machine vision gap: leading models match or exceed human accuracy on OOD distortions, while image-level consistency remains imperfect. Larger training datasets narrow that consistency gap in some cases, and the benchmark supports future comparisons.

  • Best models now match or exceed human accuracies on out-of-distribution distortion datasets.The study evaluates models differing in objective function, architecture, and training dataset size.
  • An image-level consistency gap remains between human and machine vision.The comparison concerns individual-image decisions rather than only aggregate accuracy.
  • The study focuses on non-adversarial OOD robustness and limits human presentation times to 200 ms to compare feedforward models with human perception.Human ceiling performance could be higher with more viewing time, but that would extend beyond the study’s core object-recognition comparison.
  • Increasing dataset size by one to three orders of magnitude can make models more human-like on individual-image decisions.The authors describe this result as having mixed intellectual and practical implications because large-scale training requires substantial infrastructure.
  • The open-source modelvshuman Python library enables PyTorch and TensorFlow models to be tested on the benchmark suite.The benchmark is intended to measure future progress using OOD generalisation data and human psychophysical responses.

B Error consistency predictions

The error-consistency experiments compare dataset choice with architecture, and model flexibility with architectural constraints. The reported observations support both predictions, although some dataset-specific differences are small.

  • Table 1 compares error consistency across five non-parametric datasets for dataset-versus-architecture and flexibility-versus-constraints effects.
  • Dataset vs. architecture: Training ResNet-50 on Stylized-ImageNet produces lower error consistency than comparing differently architected CNNs trained on ImageNet.The relationship is not perfect, with only a small difference for silhouette images.
  • Flexibility vs. constraints: ViT-S shows higher error consistency with ResNet-50 than with the constrained BagNet-9 model.This difference is not large for the cue-conflict dataset.

C Experimental details regarding psychophysical experiments

Participants completed a controlled visual categorisation experiment involving corrupted images and mouse responses. The protocol included response instructions, category familiarisation, and visual-acuity screening.

  • Participants viewed corrupted images and selected responses with a computer mouse.They were instructed to respond accurately and guess when unsure.
  • Participants named all 16 response categories before testing to reduce misunderstanding.
  • Visual acuity was measured with a Snellen chart to ensure normal or corrected-to-normal vision.
  • COVID-19 precautions included distancing, masking, hand disinfection, and equipment disinfection.The experimenter was tested twice weekly, and participants were informed about an adjacent cost-free testing option.

C.3 Participant remuneration

Participants received hourly payment or course credit, with an additional performance-based bonus. The consent materials described voluntary participation and data-handling conditions.

  • Participants were paid 10C per hour or granted course credit.
  • A bonus of up to 15C was awarded according to performance relative to the previous experiments’ mean accuracy.The bonus range extended from a threshold 15% below that mean to maximum payment at 15% above it.
  • The consent declaration stated that participation was voluntary and could be stopped at any time without disadvantage.
  • Contact information could be stored digitally within the Wichmannlab solely for participant contact and would not be shared with third parties.

E Training with CLIP labels

The experiments tested whether CLIP-derived labels explain CLIP’s strong performance, while comparing self-supervised and supervised models under matched augmentations. CLIP labels alone did not substantially improve standard ResNet-50 performance.

  • Evaluation: Figure 6 aggregates comparisons among standard ResNet-50, CLIP backends, and ResNet-50 models trained with CLIP hard or soft labels, with humans as a reference.Figure 16 provides detailed performance across datasets.
  • CLIP labels: Training a standard ResNet-50 with CLIP-provided labels did not strongly improve any evaluated metric.This suggests ImageNet label errors are unlikely to limit standard models’ OOD accuracy or consistency with human responses.
  • Self-supervised training: SimCLR’s superior performance was largely attributed to its data augmentation scheme rather than its contrastive self-supervised loss.The comparison used augmentation-matched supervised baseline models.

G Benchmark scores

The benchmark aggregates model performance across OOD datasets using accuracy and human-likeness rankings. Human-like behaviour is ranked by the mean rank across accuracy difference, observed consistency, and error consistency.

  • Benchmark scores: Models are ranked by out-of-distribution robustness and human-like behaviour using separate benchmark tables.Human-like behaviour receives a final ordering from the mean rank across three distinct metrics.
  • Human-like behaviour: Three metrics—accuracy difference, observed consistency, and error consistency—produce distinct model rankings.The benchmark combines their mean rank to order models by human-like behaviour.
  • Dataset coverage: Parametric and nonparametric datasets are included in benchmark-score calculations, with all datasets considered under the stated conditions.The benchmark applies dataset-specific conditions when aggregating scores.

H Regression model

Regression models examined how architecture, dataset size, and objective function relate to OOD accuracy and error consistency with humans. Dataset size was significant for both outcomes, while architecture and objective effects differed by metric.

  • Error consistency: Error consistency showed significant effects for transformers, large datasets, and the dataset-size-by-objective interaction.The corresponding p-values were 0.01936, 3.39e-05, and 0.00625, respectively.
  • Error consistency: Error consistency showed no significant main effect of objective function (p = 0.10062, n.s.).The final model reported multiple R-squared = 0.5045 and adjusted R-squared = 0.4623.
  • OOD accuracy: OOD accuracy showed a significant effect of large datasets and a significant dataset-size-by-architecture interaction.The p-values were 4.65e-09 and 0.0174, respectively.
  • OOD accuracy: OOD accuracy showed no significant main effect of transformers versus CNNs (p = 0.8553, n.s.).The final model reported multiple R-squared = 0.5848 and adjusted R-squared = 0.5588.
  • Limitations: The regression analysis is limited because linear models capture only linear effects, and the OOD-accuracy residuals were not perfectly normally distributed.Diagnostic plots indicated a better fit for the error-consistency model than for the OOD-accuracy model.

I Mapping behavioural decisions

Because human observers cannot practically classify images into 1,000 ImageNet classes, model outputs are mapped to 16 human-compatible basic categories. Benchmark figures then compare accuracy and error consistency across model types and OOD datasets.

  • Category mapping: ImageNet-trained models’ 1,000-class decision vectors are mapped to 16 basic categories using the WordNet hierarchy.The 16 categories represent 227 ImageNet classes, roughly one quarter of all ImageNet categories.
  • Category mapping: Humans classify images into 16 basic categories because assigning people 1,000 ImageNet classes is impractical and many fine-grained class names are unfamiliar.The categories include broad classes such as chair, dog, airplane, and boat.
  • Benchmark figures: Figure 9 compares model and human performance using accuracy and image-level error consistency across 17 OOD datasets.Symbols distinguish convolutional models, vision transformers, and humans.
  • Benchmark figures: Figures 10–14 sort error-consistency results with humans for sketch, stylized, edge, silhouette, and cue-conflict images.These figures extend the error-consistency comparisons across the named OOD stimulus types.
  • Model comparisons: Figure 15 compares self-supervised SimCLR models with augmentation-matched supervised baselines, using model width as a further pairing distinction.Humans are shown as red diamonds for context.
  • Model comparisons: Figure 16 compares standard ResNet-50, CLIP backends, and ResNet-50 models trained on CLIP hard or soft labels against humans.The figure uses distinct symbols and colours for architecture and label conditions.
Loading 2106.07411v2…