Source-linked AI summary
Comparing deep neural networks against humans: object recognition when the signal gets weaker
Robert Geirhos, David H. J. Janssen, Heiko H. Schütt, Jonas Rauber, Matthias Bethge, Felix A. Wichmann
TL;DR
The paper asks whether current DNNs are behaviourally similar to humans when object-recognition signals are degraded, a question motivated by limited psychophysical comparisons. It compares humans with three DNNs across several image degradations and finds greater human robustness, with increasingly divergent error patterns as signals weaken.
Problem
Behavioural comparisons between humans and DNNs remain limited, especially for stimulus-dependent accuracy and category-specific error patterns under degraded signals.
Method
The study psychophysically compares humans with AlexNet, GoogLeNet, and VGG-16 on briefly presented, backward-masked object images manipulated in colour, contrast, noise, and eidolon distortions.
Results
Humans are more robust than the tested DNNs to contrast reduction, additive noise, and novel eidolon distortions, while human–DNN classification error patterns progressively diverge as signals weaken.
Takeaways & Limitations
The findings provide a behavioural benchmark for improving DNN robustness and motivate investigation of brain mechanisms supporting human robustness to novel distortions.
Abstract
from arXiv · showhide
Human visual object recognition is typically rapid and seemingly effortless, as well as largely independent of viewpoint and object orientation. Until very recently, animate visual systems were the only ones capable of this remarkable computational feat. This has changed with the rise of a class of computer vision algorithms called deep neural networks (DNNs) that achieve human-level classification performance on object recognition tasks. Furthermore, a growing number of studies report similarities in the way DNNs and the human visual system process objects, suggesting that current DNNs may be good models of human visual object recognition. Yet there clearly exist important architectural and processing differences between state-of-the-art DNNs and the primate visual system. The potential behavioural consequences of these differences are not well understood. We aim to address this issue by comparing human and DNN generalisation abilities towards image degradations. We find the human visual system to be more robust to image manipulations like contrast reduction, additive noise or novel eidolon-distortions. In addition, we find progressively diverging classification error-patterns between humans and DNNs when the signal gets weaker, indicating that there may still be marked differences in the way humans and current DNNs perform visual object recognition. We envision that our findings as well as our carefully measured and freely available behavioural datasets provide a new useful benchmark for the computer vision community to improve the robustness of DNNs and a motivation for neuroscientists to search for mechanisms in the brain that could facilitate this robustness.
1 Introduction
Human core object recognition is rapid and robust, while DNNs offer human-level object classification but may process visual inputs differently. This motivates fair behavioural comparisons under image degradations to test whether DNNs model human recognition.
- Human visual recognition: Humans can identify objects in standard images within a single fixation in less than 200 ms, largely independent of viewpoint and orientation.This ability is termed core object recognition and is associated with the ventral visual pathway.
- DNNs and human vision: DNNs trained on millions of labeled images achieve human-level performance on natural-scene object classification and sometimes surpass human performance.Their success has encouraged comparisons between network components and visual areas in the primate brain.
- Research gap: Thorough behavioural comparisons are rare because matching overall accuracy does not establish similarity in stimulus-dependent performance or category-specific errors.Fine-grained comparisons can examine accuracy changes with image properties and progressively approach trial-by-trial behavioural prediction.
- Study aim: The study compares human observers with AlexNet, GoogLeNet, and VGG-16 using briefly presented, backward-masked images degraded by colour, contrast, noise, or eidolon distortions.The paradigm is designed to minimize feedback influence, matching the purely feedforward computations used by the networks.
- Study aim: The analyses and freely available behavioural datasets are intended as a benchmark for improving DNN robustness and as motivation to investigate mechanisms supporting human robustness.These aims concern both computer-vision model development and neuroscience research.
2 Methods
The experiments tested human and DNN object categorization under four controlled image degradations using brief presentation, backward masking, and fixed response procedures. Human participants and three pretrained networks completed matched category-recognition tasks.
- Stimuli and conditions: Images were degraded by grayscale conversion, contrast reduction, uniform white noise, or increasing eidolon distortion strength.The eidolon experiment used 24 conditions, whereas contrast and noise experiments used eight each.
- Procedure: Each trial presented a fixation square for 300 ms, an image for 200 ms, a 1/f noise mask for 200 ms, and a response screen for 1500 ms.Observers selected one of 16 entry-level categories by clicking the response screen.
- Procedure: The response screen remained visible for 1500 ms, and the last category selected during that interval counted as the answer.Trials without a human response were recorded as incorrect.
- Participants and trials: The experiments included 1,280 trials per participant, with three observers in the colour experiment and five in each other experiment.The eidolon experiment comprised three sessions of 1,280 trials each.
- Participants and trials: Participants were University of Tübingen students aged 19–28 years with normal or corrected-to-normal vision; one participant was an author.Other participants were paid or received course credit.
- DNNs: The networks were pretrained AlexNet, GoogLeNet, and VGG-16, evaluated using single feedforward passes on 224 × 224-pixel center crops.Grayscale inputs were replicated across the three RGB planes required by the networks.
3 Results
Across image degradations, humans generally retained more robust and less category-biased recognition than DNNs, while human–DNN error patterns increasingly diverged as task difficulty rose.
- 3.1.1 Colour-experiment: 4.81% average accuracy drop for DNNs versus 1.88% for humans on grayscale images.All three networks showed a significant grayscale deficit, whereas humans did not show a significant average reduction.
- 3.1.2 Contrast-experiment: At 1% contrast, VGG-16 achieved 17.5% correct responses while the other systems approached chance, and AlexNet and GoogLeNet declined faster than humans and VGG-16.At full contrast, VGG-16, GoogLeNet and human observers were approximately 91–94% accurate, while AlexNet reached 84%.
- 3.1.2 Contrast-experiment: Lower contrast caused all DNNs to bias responses toward a few categories, whereas humans largely continued distributing responses sensibly across the 16 categories.The response-distribution entropy decreased for DNNs as contrast was lowered, indicating increasingly uneven category use.
- 3.1.3 Noise-experiment: When noise width increased from 0.0 to 0.1, human accuracy fell from 80.50% to 75.13%, compared with much larger drops for every network.VGG-16 declined from 89.91% to 44.02%, GoogLeNet from 81.70% to 34.02%, and AlexNet from 70.00% to 19.29%.
- 3.1.4 Eidolon-experiment: For maximal-coherence eidolon distortions, humans reached 75.3% accuracy versus 38.3% averaged across networks at reach value 8.Network and human accuracies were similar at minimal and extreme distortion, but humans were more accurate at intermediate distortions.
- 3.2 Confusion and confusion difference matrices: As difficulty increased, DNN and human classification behavior diverged despite matched overall accuracy, with networks increasingly concentrating responses on particular categories.At noise width 0.35, AlexNet and GoogLeNet responded bottle on 92.32% and 85.71% of images, while VGG-16 responded dog on 62.50%.
4 Discussion
The discussion finds that human object recognition is more robust than current DNNs under degraded signals, while error patterns increasingly diverge as difficulty rises. It also identifies substantial differences among individual networks and points to robust representations and possible mid-level mechanisms as directions for improvement.
- Robustness: DNNs outperform human observers on non-distorted coloured images, but humans are more robust to contrast reduction, noise, and eidolon distortions.The networks perform best on the image type represented in their training data, whereas humans generalise better to several degradations.
- Colour: Human accuracy declines only marginally after grayscale conversion, whereas all three DNNs show significant performance decrements.AlexNet drops by more than 7%; VGG-16 and GoogLeNet rely less on colour but still more than the average human observer.
- Contrast: At low contrast, humans outperform AlexNet and GoogLeNet but not VGG-16, while all DNNs develop increasing category biases.The authors note that ImageNet training alone produces suboptimal contrast invariance and discuss preprocessing or contrast-varied augmentation as possible remedies.
- Model differences: The three DNNs differ considerably in their ability to handle grayscale, contrast, and noise, arguing against treating DNNs as a single model type.The discussion also proposes that robust representations generalising to novel distortions, potentially supported by depth-layering mechanisms, could improve DNN robustness.
- Noise: As noise increases, human and network confusion patterns diverge despite matched overall performance, and all networks increasingly bias decisions toward a few categories.The authors suggest that networks may incorporate noise into object texture, whereas humans may perceive it as a foreground layer.
A.1 Categories and image database
The study combines ImageNet’s large image supply with MS COCO’s entry-level category structure to create suitable object-recognition stimuli. WordNet hypernym mappings yield 16 clearly non-ambiguous categories.
- Categories and image database: The stimuli combine ImageNet images with MS COCO entry-level categories to better match how humans naturally name objects.ImageNet supplies millions of labelled images, while MS COCO provides 91 broader categories such as dog rather than German shepherd.
- Categories and image database: WordNet hypernym relationships map ImageNet labels to MS COCO categories, retaining 16 clearly non-ambiguous categories.The mapping links specific labels to broader categories when the relevant “kind of” relationship exists.
A.2 Image preprocessing
Images were standardized through cropping, resizing, controlled degradations, and specified file-format handling before human and DNN experiments.
- Image preparation: Images were filtered to remove grayscale files and undersized images, then center-cropped and downsampled to 256 × 256 pixels.The source pool consisted of ImageNet images from 16 entry-level categories.
- Contrast experiment: Contrast was manipulated across eight levels from 1% to 100% using pixelwise rescaling.The contrast conditions were c ∈ {1, 3, 5, 10, 15, 30, 50, 100%}.
- Noise experiment: Noise stimuli began at 30% contrast and received uniform white noise across eight ranges, with clipping at the strongest levels.Clipping affected 17.2% of pixels at w = 0.6 and 44.4% at w = 0.9.
- Eidolon experiment: Eidolon distortions were generated with reach, coherence, and fixed grain parameters using the PartiallyCoherentDisarray function.Grain was held constant at 10.0, corresponding to medium-grained distortion.
- Stimulus examples: Example stimuli illustrated contrast and noise conditions using bicycle, dog, and keyboard images, while separate examples showed eidolon coherence levels.The examples were drawn from the experimental image pool.
A.3 Apparatus
Stimuli were presented centrally on a calibrated high-refresh LCD monitor while a chin rest stabilized viewing position.
- Display and viewing: Stimuli appeared at the screen center at 256 × 256 pixels, corresponding to 3 × 3 degrees of visual angle from 123 cm.A VIEWPixx LCD monitor displayed the stimuli at 120 Hz in a dark chamber.
B Fine-tuning on distortions
Because the tested networks were not trained for degraded images, preliminary fine-tuning experiments examined whether condition-specific adaptation could improve robustness.
- Fine-tuning results: Fine-tuning on specific test conditions substantially improved DNN performance and could surpass humans on noisy low-contrast images.This result was reported as preliminary.
C JPEG vs. PNG
The JPEG-versus-PNG comparison found little format effect in most experiments but a systematic advantage for PNG in contrast conditions, especially at low contrast.
- Overall comparison: PNG and JPEG produced hardly any performance difference in colour, noise, and eidolon experiments.The comparison was conducted to assess possible effects of lossy JPEG compression.
- Contrast experiment: Both humans and DNNs performed better with PNG than JPEG in the contrast experiment, especially in the low-contrast regime.Human data were collected from three observers using the same images as in the original contrast experiment.
- Contrast experiment: VGG-16 showed an average 8.82 % performance benefit from PNG over JPEG in the contrast experiment.The passage reports that VGG-16 achieved better-than-human performance in this comparison, but the supplied text truncates the remainder of that result.
- Interpretation and recommendation: Low-contrast JPEG compression can introduce artefacts and alter the effective contrast level, motivating lossless image saving for future experiments.The authors recommend lossless saving because performance was equal or better with PNG for both humans and machines.
- Figure structure: The contrast figures separate JPEG and PNG accuracy and response-distribution entropy, while the eidolon figure reports accuracy and entropy across coherence conditions.The contrast figure also distinguishes high, medium, and low performance conditions in confusion-difference analyses.