Source-linked AI summary
ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, Wieland Brendel
TL;DR
Whether CNNs recognize objects through shape or texture remains unresolved, despite its importance for understanding model decisions and using CNNs as models of human vision. This paper compares humans and CNNs on texture-shape conflicts and trains ResNet-50 on Stylized-ImageNet, finding that standard CNNs are texture-biased while shape-biased models better match humans and are more robust.
Problem
Evidence conflicted over whether CNN object recognition relies primarily on global shape or local texture, an important question for understanding CNN decisions and modeling human vision.
Method
The authors compare humans and CNNs on texture-shape cue conflicts and train ResNet-50 on Stylized-ImageNet, which replaces object textures with artistic styles.
Results
ImageNet-trained CNNs strongly favor textures unlike humans, whereas shape-biased models achieve higher classification and object-recognition performance and greater robustness to image distortions.
Takeaways & Limitations
Shape-based representations offer a more plausible model of human object recognition and can benefit robust inference across recognition tasks.
Takeaways & Limitations
A substantial fraction of automatically stylized images were difficult for both humans and CNNs to recognize.
Abstract
from arXiv · showhide
Convolutional Neural Networks (CNNs) are commonly thought to recognise objects by learning increasingly complex representations of object shapes. Some recent studies suggest a more important role of image textures. We here put these conflicting hypotheses to a quantitative test by evaluating CNNs and human observers on images with a texture-shape cue conflict. We show that ImageNet-trained CNNs are strongly biased towards recognising textures rather than shapes, which is in stark contrast to human behavioural evidence and reveals fundamentally different classification strategies. We then demonstrate that the same standard architecture (ResNet-50) that learns a texture-based representation on ImageNet is able to learn a shape-based representation instead when trained on "Stylized-ImageNet", a stylized version of ImageNet. This provides a much better fit for human behavioural performance in our well-controlled psychophysical lab setting (nine experiments totalling 48,560 psychophysical trials across 97 observers) and comes with a number of unexpected emergent benefits such as improved object detection performance and previously unseen robustness towards a wide range of image distortions, highlighting advantages of a shape-based representation.
1 INTRODUCTION
The introduction frames a conflict between the view that CNNs learn object shapes and evidence that textures strongly influence their recognition. The paper tests this debate and shows that training can shift CNNs toward shape bias, improving robustness and recognition performance.
- Competing hypotheses: CNNs are widely believed to build increasingly complex object-shape representations from lower-level visual features.This intuition is supported by visualisations highlighting object parts and proposals that CNNs model human shape perception.
- Competing hypotheses: CNNs can recognise images from texture cues even when global shape is destroyed, yet perform poorly on sketches that preserve shape without texture.These findings motivate questioning whether standard CNNs rely primarily on shape for object recognition.
- Study objective: The paper uses carefully designed experiments and style transfer to resolve the contradictory hypotheses about CNN recognition strategies.The authors position this debate as relevant to both deep learning and the use of CNNs as models of human vision.
- Main contribution: Training on a suitable dataset can overcome standard CNN texture bias and shift representations toward shape bias.Networks with higher shape bias are reported to gain robustness to many image distortions and higher classification and object-recognition performance.
2 METHODS
The study compares human and CNN classification using controlled psychophysical experiments and matched image stimuli. It evaluates texture and shape biases with multiple image transformations, cue-conflict images, and a stylized ImageNet training set.
- Psychophysical procedure: Human observers viewed fixation, stimulus, and pink-noise-mask sequences in a controlled lab paradigm using the same images evaluated by CNNs.Each trial used 300 ms fixation, 200 ms stimulus presentation, and a 200 ms full-contrast pink-noise mask.
- Models and participants: 48,560 psychophysical trials from 97 participants were compared with predictions from AlexNet, GoogLeNet, VGG-16, and ResNet-50.CNN predictions from 1,000 ImageNet classes were mapped to 16 categories through the WordNet hierarchy.
- Experimental stimuli: Six major and three control experiments assessed texture and shape biases through original, greyscale, silhouette, edge, and texture image conditions.The first five experiments were simple object-recognition tasks differing in the image features available to participants.
- Cue-conflict experiment: 1280 cue-conflict images were created by iterative style transfer between texture images and object-content images, with 80 images per category.Object and texture images were selected only when all four networks classified them correctly, supporting interpretation of the cue-conflict experiment.
- Stylized-ImageNet: Stylized-ImageNet replaced each ImageNet image’s original texture with a randomly selected painting style using AdaIN style transfer at α = 1.0.The style source was Kaggle’s Painter by Numbers data set, containing 79,434 paintings.
3 RESULTS
ImageNet-trained CNNs rely more on texture than shape, unlike humans, while training the same ResNet-50 on Stylized-ImageNet shifts it toward shape-based recognition. This shift improves alignment with human behavior and yields gains in object detection and robustness to image distortions.
- Cue-conflict recognition: Humans retained higher recognition accuracy than CNNs for silhouette and edge stimuli, revealing a stronger reliance on global shape cues.Original, texture, and greyscale images were recognised correctly by both humans and CNNs, but CNN accuracy fell much more for silhouettes and edge stimuli.
- Cue-conflict recognition: 95.9% of correct human cue-conflict decisions followed the shape category, whereas CNN responses showed the opposite texture-biased pattern.The experiment used images with natural but contradicting texture and shape evidence, exposing which cue each system relied on.
- Stylized-ImageNet training: 79.0% top-5 accuracy was achieved by ResNet-50 on Stylized-ImageNet, versus 92.9% on ImageNet, indicating that removing predictive textures makes classification harder.ImageNet features generalised poorly to Stylized-ImageNet, supporting the interpretation that texture is no longer predictive there.
- Stylized-ImageNet training: 10.0% top-5 accuracy on Stylized-ImageNet versus 70.0% on ImageNet for a 9 × 9 BagNet showed that local texture features alone cannot solve Stylized-ImageNet.BagNets lack long-range spatial relationships because their receptive fields are restricted, linking their reduced Stylized-ImageNet performance to the need for shape information.
- Shape bias: Shape bias increased from 22% for an ImageNet-trained ResNet-50 to 81% after Stylized-ImageNet training, approaching human levels in many categories.The same standard ResNet-50 architecture therefore learned a substantially more shape-based representation under Stylized-ImageNet training.
- Emergent benefits: Incorporating Stylized-ImageNet improved object detection from 70.7 to 75.1 mAP50 on Pascal VOC 2007 and from 52.3 to 55.2 mAP50 on MS COCO.The SIN-trained network also outperformed the ImageNet-trained network on almost all tested image manipulations and approached human-level distortion robustness without seeing distortions during training.
4 DISCUSSION
The findings reveal that ImageNet-trained CNNs and humans can achieve similar standard-image accuracy while relying on different classification strategies. Training ResNet-50 on Stylized-ImageNet reduces texture bias by inducing shape-based representations, which improve recognition and robustness to image distortions.
- Behavioural differences: ImageNet-trained CNNs and human observers show marked behavioural differences despite similarly high accuracies on standard images.This suggests their underlying classification strategies differ, challenging the use of CNNs as models of human object recognition.
- Reducing texture bias: Stylized-ImageNet removes local cues through style transfer and forces networks beyond texture recognition.The dataset was introduced specifically to reduce CNN texture bias.
- Shape-based representations: ResNet-50 trained on Stylized-ImageNet learns object recognition based on shape, showing that CNN texture bias is induced by ImageNet training data rather than architecture.The same architecture can therefore learn a shape-based representation under different training data.
- Robustness benefits: A more shape-based representation benefits recognition tasks using pre-trained ImageNet CNNs and often reaches or surpasses human-level robustness to diverse image distortions.The Stylized-ImageNet-trained ResNet-50 was never trained on the specific image degradations.
5 CONCLUSION
The paper concludes that contemporary machine recognition overly relies on object textures rather than global shapes. It demonstrates that shape-based representations induced with Stylized-ImageNet support robust inference and provides openly available resources for further goals.
- 5 CONCLUSION: Machine recognition overly relies on object textures rather than global object shapes.This contradicts the common assumption that recognition primarily uses global object shapes.
- 5 CONCLUSION: Stylized-ImageNet can induce shape-based representations that improve robust inference.The conclusion identifies shape-based representation as advantageous for robust inference.
- 5 CONCLUSION: The authors provide openly available model weights, code, and behavioural data from 49K trials across 97 observers.These resources are intended to support the paper’s stated goals.
A APPENDIX · A.2 PROCEDURE
The procedure used a brief masked-image categorisation paradigm with 16 response categories, following Geirhos et al. (2018) for maximal comparability. Participants were familiarised with all categories and selected the category they believed was presented.
- A.2 PROCEDURE: Trials began with a 300 ms fixation square, followed by a 200 ms stimulus image.The stimulus was immediately followed by masking.
- A.2 PROCEDURE: A full-contrast pink-noise mask with a 1/f spectral shape covered the stimulus area for 200 ms.The mask was the same size as the stimulus image.
- A.2 PROCEDURE: Participants chose among 16 entry-level categories using a response screen displayed for 1500 ms.The category icons were arranged in a 4 × 4 grid.
- A.2 PROCEDURE: Before each experiment, participants viewed the response screen and named all 16 categories.This preparation provided an overview of the possible stimulus categories and checked that they were clear from the beginning.
- A.2 PROCEDURE: Participants were instructed to click the category they believed had been presented.Responses were entered through the response screen.
- A.2 PROCEDURE: Responses could be changed during the 1500 ms response interval, and only the last entered response counted.The procedure therefore recorded the final selection made within the response window.
A.3 APPARATUS
Observers viewed 224×224-pixel stimuli foveally on a high-refresh-rate LCD monitor in a dark cabin. Stimuli occupied 3×3° of visual angle at a 107-cm viewing distance against a standardized grey background.
- A.3 APPARATUS: 224×224-pixel stimuli were displayed on a 22-inch, 120-Hz VIEWPixx LCD monitor in a dark cabin.The monitor had a 484×302-mm screen and 1920×1200-pixel resolution.
- A.3 APPARATUS: Stimuli were presented foveally at the screen center, subtending 3×3° of visual angle at a 107-cm viewing distance.The background grey value was 0.7614 in the [0, 1] range, matching the average greyscale value of the original-experiment stimuli.
A.4 PARTICIPANTS
The study included 97 human observers, each participating in only one experiment and reporting normal or corrected-to-normal vision.
- Participants: 97 human observers participated in the study.Their distribution across experiments is summarized in Table 3.
- Participants: No observer participated in more than one experiment.
- Participants: All participants reported normal or corrected-to-normal vision.
A.5 CNN MODELS & TRAINING DETAILS
The study evaluated multiple CNN architectures and detection models using ImageNet-pretrained networks, with ResNet-50 serving as the standard architecture for ImageNet and Stylized-ImageNet comparisons. Additional evaluations used BagNets, Faster R-CNN, pretrained classification networks, and SIN-trained AlexNet and VGG-16.
- ResNet-50: ResNet-50 used the standard torchvision.models implementation; ImageNet results came from unmodified pretrained weights, while SIN results used the same architecture.ImageNet pretraining was used throughout experiments because it empirically produced better results across experiments and models.
- BagNets: BagNets used ImageNet-pretrained weights and architectures from Brendel and Bethge, with SIN training matched to ResNet-50 for comparability.The matched SIN protocol trained for 60 epochs with SGD and identical hyperparameters.
- Faster R-CNN: Faster R-CNN used default hyperparameters, a replaced ResNet-50 or ResNet-152 encoder, and custom input whitening; Pascal VOC 2007 training lasted 7 epochs.Training used batch size 1, learning rate 0.001, and decay after epoch 5.
- Pretrained classification models: Figure 4 evaluated pretrained AlexNet, GoogLeNet, and VGG-16 models using Caffe and ImageNet-trained weights from the cited model repositories.The models were sourced from the Caffe Model Zoo or the Oxford VGG-16 repository.
- SIN-trained classification models: AlexNet and VGG-16 were trained on SIN under ResNet-50 conditions, except for learning rates of 0.001 and 0.01, respectively, with decay after epochs 20 and 40.Both networks were trained for 60 epochs.
A.6 IMAGE MANIPULATIONS AND IMAGE DATABASE
The study used nine image experiments built from standardized 224 × 224 PNG stimuli, including original, texture, cue-conflict, silhouette, edge, greyscale, and distorted-image conditions. Cue-conflict stimuli combined texture and shape information through style transfer or texture-filled silhouettes, while robustness stimuli applied established distortions to natural images.
- Image manipulations: Greyscale stimuli reused the original images after conversion with skimage.color.rgb2gray, and CNN inputs repeated the greyscale channel three times.Silhouette stimuli showed entirely black objects on white backgrounds, whereas edge stimuli used MATLAB’s Canny edge extractor.
- Cue-conflict stimuli: 1,280 cue-conflict images were generated by iterative style transfer between texture and content images, with 80 images per category.Although 48 × 160 = 7680 images were possible, the study used a balanced subset of 1,280 images.
- Cue-conflict stimuli: Filled-silhouette stimuli created cue conflicts by masking texture images with object silhouettes, retaining texture within black silhouette pixels and white backgrounds elsewhere.The texture database was expanded by rotating textures through ten previously chosen angles uniformly distributed between 0 and 360 degrees.
- Robustness stimuli: Robustness evaluations applied the image manipulations used by Geirhos et al. (2018) to the same natural images, with human reference accuracies taken from that study.Human error bars represented the full range of results for human observers.
A.7 STYLIZED-IMAGENET (SIN) … A.10 IMAGE RIGHTS & ATTRIBUTION
The supplementary sections describe how Stylized-ImageNet was generated, test human cue-conflict responses under explicit instructions and alternative stimulus construction, and document image attribution. Humans retain a strong default shape bias, while SIN-trained ResNet-50 responds more to shape than its ImageNet-trained counterpart.
- A.7 STYLIZED-IMAGENET (SIN): Stylized-ImageNet was created by applying AdaIN style transfer once to every ImageNet training and validation image.The implementation used the specified PyTorch AdaIN repository and Painter by Numbers images as style sources.
- A.8 RESULTS: CUE CONFLICT CONTROL EXPERIMENTS (DIFFERENT INSTRUCTIONS): The supplementary comparisons include accuracy, PASCAL VOC 2007 mAP50 object detection, ImageNet-C corruption error, and five no-cue-conflict experiments for IN versus SIN training.The reported tables and figure captions define mCE as the mean of 15 corruption-error values and distinguish SIN, IN, and fine-tuning.
- A.8 RESULTS: CUE CONFLICT CONTROL EXPERIMENTS (DIFFERENT INSTRUCTIONS): Under shape-focused instructions, human responses were almost identical to neutral-instruction results, indicating that observers defaulted to using shapes.This control tested whether the main cue-conflict findings depended on neutral instructions.
- A.8 RESULTS: CUE CONFLICT CONTROL EXPERIMENTS (DIFFERENT INSTRUCTIONS): Even when instructed to ignore shapes and select textures, human observers retained a substantial shape bias, showing that overriding shape use was difficult.The instruction experiments compared explicit texture- and shape-directed responses with the neutral condition.
- A.9 RESULTS: FILLED SILHOUETTE EXPERIMENT: A filled-silhouette control using texture images cropped with an object-shaped mask also produced a human shape bias, reducing dependence on style-transfer stimulus generation.The alternative stimuli paired an object silhouette with texture content and were visualized in Figure 7.
- A.9 RESULTS: FILLED SILHOUETTE EXPERIMENT: ResNet-50 trained on SIN responded with the shape category more often than ResNet-50 trained on IN in the filled-silhouette experiment.The authors caution that these stimuli lack the clear-cut texture-shape distinction of style-transfer cue conflicts, but the results broadly agree with the main findings.
- A.10 IMAGE RIGHTS & ATTRIBUTION: Image sources used for Figure 7 were documented with their URLs, creators, and applicable license terms, including a CC0 license for the top-row cat image.The attribution terms also apply to reused images appearing in Figures 1, 2, and 9.
- A.10 IMAGE RIGHTS & ATTRIBUTION: Additional comparisons examined texture bias beyond ImageNet-trained networks, including ResNet-101 trained on ImageNet versus Open Images Dataset V2 and very deep, wide, and compressed networks.Figure 13 also includes human data for comparison.