Source-linked AI summary
A Study and Comparison of Human and Deep Learning Recognition Performance Under Visual Distortions
Samuel Dodge, Lina Karam
TL;DR
Image-quality distortions such as blur and noise challenge DNN classification despite strong clean-image performance. The paper compares 15 human subjects with DNNs under controlled distortions and finds that humans remain substantially more accurate, with little overlap in their errors. The results motivate more robust learning systems while remaining bounded by the study’s limited image and class coverage.
Problem
The study asks whether DNNs that match or exceed humans on clean images also retain comparable performance under blur and noise distortions.
Method
The paper compares DNN classification with 15 human subjects on ImageNet dog images distorted by controlled Gaussian blur and additive Gaussian noise.
Results
Humans achieve much higher accuracy than DNNs on distorted images, while human and DNN misclassifications generally show little significant correlation.
Takeaways & Limitations
The comparison suggests that human visual representations may be more robust and could guide research on more robust learning systems.
Takeaways & Limitations
The study uses a limited number of images and classes, which could make error-correlation estimates less reliable.
Abstract
from arXiv · showhide
Deep neural networks (DNNs) achieve excellent performance on standard classification tasks. However, under image quality distortions such as blur and noise, classification accuracy becomes poor. In this work, we compare the performance of DNNs with human subjects on distorted images. We show that, although DNNs perform better than or on par with humans on good quality images, DNN performance is still much lower than human performance on distorted images. We additionally find that there is little correlation in errors between DNNs and human subjects. This could be an indication that the internal representation of images are different between DNNs and the human visual system. These comparisons with human performance could be used to guide future development of more robust DNNs.
I. INTRODUCTION
DNNs perform strongly on clean images, but distortions raise questions about whether their performance remains comparable to human recognition. The study compares humans and DNNs under blur and noise, finding a substantial human advantage and weak error correspondence.
- Motivation: 3.57% top-5 error was reported for a state-of-the-art neural network on ImageNet, compared with 5.1% for humans.These figures establish that DNNs can match or exceed human performance on good-quality images.
- Motivation: Image distortions can arise during acquisition or transmission, making robustness relevant for practical machine-learning services.Examples include low-light sensor noise, motion blur, packet loss, and missing image frequencies.
- Research Question: The study asks whether networks still match or exceed human performance when images are distorted, or whether humans retain an advantage.The question focuses on additive noise and blur, which particularly degrade neural-network performance.
- Approach: 15 human subjects classify images with varied Gaussian noise and blur levels, enabling direct comparison with DNN predictions.The experiments also examine correlations between human and network errors.
- Findings: Humans classify blurred and noisy images more accurately than DNNs, while high-distortion errors show relatively low correlation.The authors suggest this may indicate different internal models between DNNs and the human visual system.
A. Related Works
Prior work established both human advantages on some altered-image tasks and DNN superiority on clean benchmarks. This motivates testing whether fine-tuning and other robustness strategies close the human–DNN gap under natural quality distortions.
- Earlier Comparisons: Earlier studies found machine-learning classification below human performance on several synthetic or jumbled-image tasks.These tasks either used synthetic spatial arrangements or destroyed global image information through patch permutation.
- DNN Progress: Recent ImageNet experiments report that state-of-the-art DNNs achieve lower human classification error on clean images.This shift makes distorted-image recognition a more difficult comparison problem.
- Human Robustness: Humans can recognize low-resolution, low-resolution-face, and noisy-face images with useful accuracy, although noise impairs face recognition.These findings indicate that humans retain recognition ability under substantial visual degradation.
- DNN Robustness: Deep networks perform poorly on distorted images, with noise and blur producing especially strong degradation.This contrasts with their clean-image performance.
- Robustness Methods: Fine-tuning improves performance on blurred and noisy images, but models fine-tuned for one distortion do not easily generalize to others.Other approaches include distortion-dependent gating and preprocessing modules.
- Study Motivation: The study experimentally compares human recognition with DNN recognition to assess whether their performance drop under quality distortions is similar.The comparison is intended to provide insight into building more robust deep networks.
A. Dataset
The study uses a manageable, difficult subset of ImageNet consisting of related dog classes, then applies controlled Gaussian blur and additive Gaussian noise across multiple severity levels.
- Dataset Selection: The experiment limits the benchmark to reduce human-study time and subject fatigue compared with a 1000-class ImageNet evaluation.The authors note that the reduced scale creates experimental convenience but limits dataset breadth.
- Dataset Selection: The benchmark uses 10 related dog classes selected from ImageNet because their visual similarity makes recognition difficult.The selected classes include border collie, Eskimo dog, German shepherd, and seven other dog breeds.
- Distortions: Gaussian blur is tested at 5 standard deviations from 2 to 10, while additive Gaussian noise uses 5 standard deviations from 40 to 200.The noise range extends higher than earlier experiments because preliminary testing found high human accuracy at standard deviation 100.
- Distortions: Figure 2 presents samples from all 10 classes alongside corresponding images with three levels of noise and three levels of blur.The examples illustrate the stimulus families used in the study.
- Data Split: The image split contains 250 training images, 50 validation images, and 200 testing images across the 10 classes.Per class, the split uses 25 training, 5 validation, and 20 testing images.
B. Human Experiments
Human subjects first study every training image, then validate on clean images before classifying distorted test images. The protocol controls guessing, memory effects, ordering, and interface access.
- Training: Subjects must view every training image from each class before continuing to later experiment stages.Figure 3 depicts free viewing of the training set with continuation disabled until all images have been viewed.
- Validation: Clean validation images are shown randomly, and subjects must choose among the 10 class labels without a “don’t know” option.Validation performance establishes a human baseline and supports removal of poorly performing outlier subjects.
- Testing: Testing presents each new image first at the highest distortion, then at reduced distortion levels only when necessary.This design assumes correct recognition at high distortion implies correct recognition at lower distortion for the same image.
- Testing: Images are shown in random order and training examples remain available during classification to reduce memory effects and test recognition rather than memorization.Subjects choose a class after viewing the distorted image and, if desired, associated training examples.
- Participants: The study recruits 15 subjects through Amazon Mechanical Turk and requires a sufficiently large browser window for the complete interface.The window restriction also prevents subjects from taking the experiment on mobile devices.
C. Deep Neural Networks
The study evaluates pretrained deep networks on dog classification, with and without fine-tuning for Gaussian blur and additive noise. Networks are tested using a distortion procedure designed to avoid giving human subjects an advantage.
- Network architectures: The study considers VGG16, Google Inception v3, and 50-layer ResNet architectures pretrained on all 1000 ImageNet classes.Their final layers are replaced with 10-unit layers for the dog-classification task.
- Network training: The networks are fine-tuned separately on noisy or blurred data while retaining half of each training batch as clean images.Distortion levels are sampled uniformly from zero to the maximum level during fine-tuning.
- Testing procedure: Network testing begins at the highest distortion level and repeats misclassified images at reduced levels.Correct classification at a high distortion level is taken to imply correct classification at smaller distortion levels.
- Distorted-image performance: Human subjects outperform deep neural networks on blurred and noisy images, including networks fine-tuned on the respective distortions.Figure 5 compares average accuracy on distorted images.
III. RESULTS
Human subjects achieve strong clean-image accuracy, but remain more accurate than deep networks under distortions. Network errors also show limited correlation with human errors, while image difficulty varies across humans and networks.
- Clean validation: 92.9 ± 5.2% was the subjects’ average classification accuracy on clean validation images.One subject with 8% validation accuracy was removed from the analysis.
- Distorted-image accuracy: Fine-tuning increases deep-network accuracy, but accuracy remains much lower than human performance on distorted images.Human results are relatively consistent despite the sample size of 15 subjects.
- Error patterns: At high distortion levels, original networks tend to classify images into only one or two classes.Fine-tuned networks show this concentration problem to a lesser degree.
- Error correlation: Human and DNN misclassifications generally show no significant degree of correlation.The correlation is computed from confusion matrices after removing the diagonal.
- Error correlation: Low-distortion error correlations may be unreliable because few human-subject images are misclassified.The limitation arises from the low number of human errors available for the calculation.
- Image difficulty: Some images are correctly classified by both humans and networks under heavy distortion, while others favor one group or are difficult for both.The authors caution that conclusions from these examples are difficult to draw.
IV. DISCUSSION AND CONCLUSION
Fine-tuning improves DNN performance on distorted images, but humans remain substantially more accurate, and the study identifies robustness and representation differences as open issues.
- Fine-tuning improves network performance under distorted data, but accuracy remains well below human performance.
- Human visual representations seem more robust than those of state-of-the-art neural networks under visual distortions.The authors suggest that recognition under noise requires global information, whereas convolutional filters may rely on textures.
- Human and DNN misclassifications generally show little significant correlation across distortion levels.The correlation is computed from confusion matrices after removing their diagonal entries.
- The study is limited by its number of images and classes, although a larger experiment could improve error-correlation estimates.