Source-linked AI summary
Are Convolutional Neural Networks or Transformers more like human vision?
Shikhar Tuli, Ishita Dasgupta, Erin Grant, Thomas L. Griffiths
TL;DR
The paper asks whether CNNs and Vision Transformers use visual categorization strategies like humans, since accuracy alone does not reveal which stimuli systems misclassify. It compares their error consistency, shape bias, and performance on diagnostic data, finding that Transformers are more consistent with human errors and have higher shape bias than CNNs.
Problem
Accuracy does not reveal whether two decision makers use the same strategy, because systems can differ in which stimuli they misclassify and how they make errors.
Method
The study compares ImageNet-pretrained Vision Transformers and ResNets with human behavior using error-consistency metrics and the Stylized ImageNet diagnostic dataset.
Results
Vision Transformers are more consistent with human errors and have higher shape bias than traditional CNNs on the evaluated visual categorization analyses.
Takeaways & Limitations
The results support using more granular error analyses to study human-like vision models and visual object recognition.
Takeaways & Limitations
The paper notes that additional comparisons among Transformer architectures and broader uses of the JS metric remain to be explored.
Abstract
from arXiv · showhide
Modern machine learning models for computer vision exceed humans in accuracy on specific visual recognition tasks, notably on datasets like ImageNet. However, high accuracy can be achieved in many ways. The particular decision function found by a machine learning system is determined not only by the data to which the system is exposed, but also the inductive biases of the model, which are typically harder to characterize. In this work, we follow a recent trend of in-depth behavioral analyses of neural network models that go beyond accuracy as an evaluation metric by looking at patterns of errors. Our focus is on comparing a suite of standard Convolutional Neural Networks (CNNs) and a recently-proposed attention-based network, the Vision Transformer (ViT), which relaxes the translation-invariance constraint of CNNs and therefore represents a model with a weaker set of inductive biases. Attention-based networks have previously been shown to achieve higher accuracy than CNNs on vision tasks, and we demonstrate, using new metrics for examining error consistency with more granularity, that their errors are also more consistent with those of humans. These results have implications both for building more human-like vision models, as well as for understanding visual object recognition in humans.
Introduction
The paper asks whether CNNs or Vision Transformers produce classification behavior more like humans, moving beyond accuracy to compare error patterns. It situates this question in diagnostic analyses of shape-versus-texture bias and error consistency.
- CNNs are widely used across computer vision tasks, including detection, classification, segmentation, facial recognition, and captioning.
- Diagnostic datasets can reveal whether models rely more on shape or texture by applying one class’s texture to another class’s image while preserving shape.Stylized ImageNet is one such dataset.
- Transformers apply learned attention rather than the local spatial inductive bias imposed by convolutions.Their use in vision has outperformed state-of-the-art CNNs in prior work.
- This paper compares a Vision Transformer, standard CNNs, and humans on visual categorization to assess which model has more human-like classification behavior.
Convolution vs. Attention
CNNs impose translation invariance through shared convolutional weights, whereas Transformers use flexible self-attention to allocate relevance contextually. Transformers can emulate local receptive fields while also integrating information across varying distances.
- CNN convolutional layers process image patches with shared weights, hard-coding translation invariance.This resembles the response of visual-cortex neurons to specific stimuli.
- CNN local connectivity can discard global context, creating a limitation of their spatially constrained processing.
- Transformer self-attention contextually upweights relevant information instead of requiring fixed convolutional processing.
- Multi-head self-attention can implement local receptive fields like convolution while retaining greater flexibility.Transformers are not bound to use convolutions and can attend over varying distances.
Measuring Error Consistency
The paper develops increasingly granular measures of error consistency and compares CNNs, Vision Transformers, and humans on Stylized ImageNet. ViTs are generally more human-like than CNNs, although the full inter-class error distribution yields a contrasting result.
- Motivation: Accuracy alone misses whether two systems fail on the same stimuli, motivating error-consistency measures that compare their decisions.Systems can have similar accuracy while differing in which images they misclassify.
- Existing measures: Observed error overlap counts trials on which two systems are both correct or both incorrect, but it increases with accuracy and therefore includes chance agreement.Cohen’s κ normalizes observed overlap using the overlap expected from the systems’ accuracies.
- Existing measures: Cohen’s κ ignores which incorrect class a system predicts, limiting interpretation of the source of cross-system similarities and differences.Confusion matrices retain true-versus-predicted class information, but their high dimensionality and sparse population make human comparison difficult.
- JS-distance measures: Lower JS distance indicates higher error consistency and, unlike Cohen’s κ, is unaffected by overall classifier accuracy.The measures are computed from normalized error distributions and Jensen–Shannon distance.
- JS-distance measures: Class-wise JS distance compares distributions of misclassified true classes, while inter-class JS distance compares the full distribution of what classes were misclassified as what.The inter-class representation contains 240 error types, providing finer-grained information than class-wise comparisons.
- Model comparison: On Stylized ImageNet, ViT is more human-like than CNNs under Cohen’s κ and class-wise JS distance, but its inter-class JS distance is higher than ResNet’s.Across models, lower class-wise JS distance is associated with higher SIN accuracy and higher shape bias, while class-wise JS distance is not significantly tied to training-task accuracy.
Shape Bias
Shape bias measures whether models favor object shape over texture when either cue is correctly predicted. ViT has higher shape bias than traditional CNNs, and both models increase shape bias after fine-tuning.
- Shape bias is the percentage of correct shape predictions among trials where either shape or texture is predicted correctly.The analysis evaluates models on Stylized ImageNet.
- ViT has a higher shape bias than traditional CNNs on the Stylized ImageNet evaluation.This aligns partly with humans’ primary reliance on shape rather than texture.
- Both ResNet and ViT increase their shape bias after fine-tuning.The comparison is shown in Figure 7, whose vertical lines indicate averages.
Fine-tuning with Augmented Data
The study fine-tunes ViT and ResNet on augmented ImageNet data to examine how training data changes representations, shape bias, human-like errors, and accuracy. Augmentation increases shape bias in both models, while ViT largely preserves accuracy and ResNet loses more accuracy.
- Training data and augmentations substantially influence the shape and texture biases learned by vision models.Prior work found that naturalistic augmentations decrease texture bias, while random crops increase it in ImageNet-trained CNNs.
- The study fine-tunes the smallest ViT and ResNet models on ImageNet augmented with rotations, cutout, Sobel filtering, blur, color distortion, and noise.The augmentations are applied to ImageNet before fine-tuning.
- Fine-tuning makes ResNet less human-like in error consistency, whereas ViT shows no significant change and trends toward improved error consistency.For ResNet, differences are significant for Cohen’s κ and inter-class JS distance, while class-wise JS shows a non-significant trend.
- Both ResNet and ViT increase their shape bias after fine-tuning.These results are reported in Figure 7.
- Training on augmented data slightly decreases ImageNet accuracy, with a more significant decrease for ResNet than for ViT.The accuracy decrease accompanies increased shape bias.
Conclusion
The paper compares CNNs and ViTs with humans using error consistency and shape-bias analyses, including new metrics and augmented-data fine-tuning. Transformers are more human-consistent overall in the reported comparison, while both architectures gain shape bias after fine-tuning.
- The paper compares Vision Transformers, standard CNNs, and humans on visual categorization using error consistency, shape bias, and accuracy.The analysis introduces metrics beyond Cohen’s κ to examine error consistency more granularly.
- Transformers outperform CNNs in image-classification accuracy, have higher shape bias, and are largely more consistent with human errors.These are the paper’s reported cross-model conclusions.
- Fine-tuning on augmented datasets increases shape bias in both CNNs and Transformers, while Transformers maintain their accuracy.The paper reports equivalent gains in shape bias compared with CNNs.
- Further experiments could test Transformer variants, concept-level error similarity, and human-error consistency as a training loss.The authors suggest these directions could help develop more brain-like or human-like models.