PaperlayerView exact revision on arXiv

Source-linked AI summary

Informative Dropout for Robust Representation Learning: A Shape-bias Perspective

Baifeng Shi, Dinghuai Zhang, Qi Dai, Zhanxing Zhu, Yadong Mu, Jingdong Wang

arXiv:2008.04254v1cs.LGcs.CVstat.ML

TL;DR

The paper asks whether alleviating CNN texture bias can improve robustness across different settings. It proposes InfoDrop, which uses local self-information to guide a Dropout-like method, and reports improved shape bias and robustness. The authors leave the proper relationship between shape and texture for future exploration.

  • Problem

    The paper addresses whether reducing CNN texture bias can improve robustness across different settings.

    Source: §1 · p. 2, §5 · p. 9

  • Method

    InfoDrop uses local self-information to guide a Dropout-like method that decorrelates model outputs from local texture.

    Source: §3.1 · p. 3, §5 · p. 9

  • Results

    InfoDrop improves shape bias and robustness across multiple settings, and the authors report that it can be combined with other algorithms for higher robustness.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The work presents a unified approach to different kinds of robustness and examines the relationship between shape bias and robustness.

    Source: §5 · p. 9

    The proper relationship between shape and texture, including whether they act as separate or hierarchical cues, is left for future exploration.

    Source: §5 · p. 9

Abstract

from arXiv · show

Convolutional Neural Networks (CNNs) are known to rely more on local texture rather than global shape when making decisions. Recent work also indicates a close relationship between CNN's texture-bias and its robustness against distribution shift, adversarial perturbation, random corruption, etc. In this work, we attempt at improving various kinds of robustness universally by alleviating CNN's texture bias. With inspiration from the human visual system, we propose a light-weight model-agnostic method, namely Informative Dropout (InfoDrop), to improve interpretability and reduce texture bias. Specifically, we discriminate texture from shape based on local self-information in an image, and adopt a Dropout-like algorithm to decorrelate the model output from the local texture. Through extensive experiments, we observe enhanced robustness under various scenarios (domain generalization, few-shot classification, image corruption, and adversarial perturbation). To the best of our knowledge, this work is one of the earliest attempts to improve different kinds of robustness in a unified model, shedding new light on the relationship between shape-bias and robustness, also on new approaches to trustworthy machine learning algorithms. Code is available at https://github.com/bfshi/InfoDrop.

1. Introduction

The introduction links CNN texture bias with concerns about robustness and asks whether reducing that bias can improve robustness across settings. It presents InfoDrop as a lightweight, model-agnostic approach and reports consistent improvements across several robustness tasks.

1. Introduction

  • CNN vulnerability is contrasted with human visual robustness to domain gaps and noisy inputs.The introduction also describes CNN texture bias as favoring texture over shape in classification.Source: §1 · p. 1
  • The paper asks whether texture bias is a common reason for CNN non-robustness to distribution shifts, adversarial perturbations, and image corruption.CNNs are described as relying on local texture rather than shape when making decisions.Source: §1 · pp. 1–2
  • InfoDrop uses local self-information to identify texture and shape, then applies a Dropout-like method to decorrelate model outputs from local texture.Regions with less information have their corresponding output neurons zeroed with higher probability.Source: §1 · p. 2
  • Earlier shape-bias approaches are described as vulnerable to complex patterns or as computationally costly and reliant on auxiliary tasks.The introduction contrasts these approaches with InfoDrop's lightweight, model-agnostic design.Source: §1 · p. 2

1.1. Our Contribution

  • InfoDrop consistently improves robustness over baselines across domain generalization, few-shot classification, random corruption, and adversarial robustness.The authors also report that it can be combined with other algorithms to obtain higher robustness.Source: §1 · p. 2, §1.1 · p. 2

2. Related Work

Related work describes CNN vulnerability to distribution shifts, corruptions, and perturbations, alongside evidence of texture bias. Prior work connects texture bias and robustness in specific settings, while this paper frames their relation across robustness types as an open question.

2.1. Vulnerability of CNNs

  • CNNs struggle with distribution shifts, corruptions, and adversarial perturbations, motivating evaluations across few-shot classification, domain generalization, and robustness tests.The cited work includes benchmarks spanning 18 types of random corruption.Source: §2.1 · p. 2

2.2. Texture-bias of CNNs

  • Recent findings indicate that local texture can support CNN classification while shape and contour information remain difficult for CNNs to use.The literature also reports failures to transfer between images with similar shapes but different textures.Source: §2.2 · p. 3

2.3. Relation between Non-robustness and Texture-bias

  • Prior studies associate shape bias with improved robustness in particular settings, but the cited work had not examined the relationship across different non-robustness types in a unified model.Examples include adversarial training, stylized-image training, jigsaw puzzles, and suppressing local predictive power or superficial statistics.Source: §2.3 · p. 3

2.4. Bias in Human Vision

  • Human-vision research motivates InfoDrop through evidence on both shape bias and self-information-guided saliency.The cited work describes shape as an important clue in human vision and self-information as a model of saliency.Source: §2.4 · p. 3

3. Methodology

The methodology computes convolutional activations and introduces InfoDrop to reduce reliance on low-information input regions. Its design uses neighborhood-based self-information to guide neuron dropout, with stated computational cost below that of the convolutional operation.

3.1. Informative Dropout

  • InfoDrop assigns higher dropout probability to output neurons linked to lower-information input patches, aiming to decorrelate layer outputs from texture.The method uses a Dropout-like process; low-information patches are associated with repeated patterns, while local shapes are described as more informative.Source: §3.1 · pp. 3–4
  • Local self-information is used to guide filtering, with lower information increasing a neuron's likelihood of being dropped.The paper models neighborhood patches as realizations from the same distribution; this is an assumption used in its estimation approach.Source: §3.1 · pp. 3–4
  • During training, InfoDrop samples output neurons for removal, with r0 controlling the number sampled; sampling is with replacement.The expected actual dropout rate is lower than r0 because sampled neurons can collide.Source: §3.1 · p. 4

3.2. Computational Complexity

  • InfoDrop's stated complexity is O((cℓ−1 + cℓ) · hℓ · wℓ), compared with O(cℓ−1 · cℓ · hℓ · wℓ) for convolution.The analysis assumes constant kernel size and neighborhood scale, and notes spatial shape often stays unchanged through convolution.Source: §3.2 · p. 4

4. Experiments

Experiments examine InfoDrop’s shape bias and test robustness across domain generalization, few-shot classification, image corruption, and adversarial attacks. Results indicate gains across these settings, with important qualifications: severe blur and large adversarial perturbations remain challenging, and the best balance of shape and texture varies by task.

4.1. Shape-bias of InfoDrop

  • Self-information highlights medium-frequency edges and corners over repetitive high-frequency textures, while InfoDrop’s saliency and style-transfer results also favor object shape.Patch shuffling further indicates greater shape bias: InfoDrop performance declines as global structure is disrupted, whereas regular CNNs are barely affected.Source: §4.1 · pp. 5–6, Figure 3 · p. 5, Figure 4 · p. 6

4.2.1. DOMAIN GENERALIZATION

  • InfoDrop improves single-source domain-generalization performance, especially in settings involving sketch, while preserving source-domain performance; multi-source gains are consistent across target domains.The vanilla baseline with InfoDrop is better than or comparable with other state-of-the-art methods on each multi-source target domain, and the comparison with stylized-image pretraining finds both approaches improve robustness.Source: §4.2.1 · p. 6, §4.2.5 · p. 8

4.2. Robustness of InfoDrop

  • InfoDrop improves robustness across domain generalization, few-shot classification, most tested corruptions, and low-norm adversarial attacks; it also adds robustness to adversarial training.The gains are not universal across conditions: blur can further degrade performance, and large perturbations still defeat the model.Source: §4.2.1 · p. 6, §4.2.2 · p. 7, §4.2.3 · p. 8

4.2.2. FEW-SHOT CLASSIFICATION

  • InfoDrop improves ProtoNet few-shot classification under almost all tested settings and improves robustness across ProtoNet, MatchingNet, and RelationNet baselines.Gains are larger in 5-shot than 1-shot settings; the authors suggest shape features may be less discriminative than texture and require more examples.Source: §4.2.2 · p. 7

4.2.3. ROBUSTNESS AGAINST IMAGE CORRUPTION

  • On Caltech-256, InfoDrop improves baseline robustness against most tested corruptions, including noise, weather, and digital corruptions, but performance may further degrade under blur.Combining InfoDrop with adversarial training provides even better corruption robustness with little overhead.Source: §4.2.3 · pp. 7–8, Table 6 · p. 8

4.2.4. ADVERSARIAL ROBUSTNESS

  • On CIFAR-10, InfoDrop improves baseline robustness to low-norm PGD attacks but still fails under large perturbations; combined with adversarial training, it adds 1% accuracy at ℓ∞=8/255.The evaluation uses 20 PGD runs for training and testing.Source: §4.2.4 · p. 8, Table 7 · p. 8

4.3. Ablation Studies

  • Temperature T=1 gives the highest accuracy in 5-way 1-shot mini-ImageNet→CUB, while applying InfoDrop to the first residual block performs best among tested layer placements.Infinite temperature acts like regular Dropout; dropping higher-layer abstract features may degrade performance.Source: §4.3 · p. 9, Table 9 · p. 9, Table 10 · p. 9

4.4. Is Shape Information All You Need?

  • Domain-generalization performance first rises and then falls as shape bias increases, with different target domains favoring different levels; the authors conclude models need a task-dependent balance of shape and texture.Patch-shuffled classification error is used as the shape-bias indicator.Source: §4.4 · p. 9, Figure 5 · p. 9

5. Conclusion

InfoDrop uses local self-information to reduce CNN texture bias and improve shape bias and robustness across multiple settings. The paper leaves the proper relationship between shape and texture for future exploration.

5. Conclusion

  • InfoDrop detects texture and shape using local self-information, then uses a Dropout-like algorithm to decorrelate model output from local texture.The authors report improved shape bias and robustness across various settings.Source: §5 · p. 9
  • The authors report that InfoDrop can be combined with other algorithms, such as adversarial training, to achieve higher robustness.Source: §5 · p. 9
  • Whether shape and texture should act as parallel cues or hierarchically, with shape guiding coarse recognition before texture details, remains open.The authors leave this relationship for further exploration.Source: §5 · p. 9

A.1. Results on Domain Generalization and Comparison with Other Shape-biased Models

With vanilla AlexNet, InfoDrop is better than or comparable to other state-of-the-art methods across four domains. The absolute single-source domain-generalization accuracies are reported separately in Table 13.

A.1. Results on Domain Generalization and Comparison with Other Shape-biased Models

  • With vanilla AlexNet as the baseline, InfoDrop is better than or comparable to other state-of-the-art methods on all four domains.The comparison follows the setting in Wang et al. (2019a); JiGen, HEX, and PAR explicitly train shape-biased models.Source: §A.1 · p. 14
  • Among the compared methods, JiGen, HEX, and PAR explicitly train a shape-biased model.Source: §A.1 · p. 14
  • Absolute single-source domain-generalization accuracies are provided in Table 13 because Table 2 had limited space.Source: §A.1 · p. 14

A.2. Robustness Against Image Corruption

InfoDrop improves robustness against most image corruptions on both vanilla and adversarially trained CNNs. Visualizations also show saliency emphasizing global structure and self-information patterns retaining structure across stylized images.

A.2. Robustness Against Image Corruption

  • InfoDrop non-trivially improves robustness against most image corruptions on both vanilla CNN and adversarially trained CNN baselines.Source: §A.2 · p. 14

A.3. Visualization of Saliency Map

  • InfoDrop saliency is more biased toward global structure than vanilla CNN saliency, which the authors describe as more human-aligned and interpretable.Source: §A.3 · p. 14

A.4. Visualization of Self-Information

  • Across differently stylized images, self-information distributions mostly retain their pattern, accentuating global structure and suppressing local texture.In contrast, edge detection is strongly affected by style and can severely ruin image content.Source: §A.4 · p. 14

B. Experimental Settings

The experiments tune r0 and T on validation data, fix h and R, and vary where InfoDrop is applied in ResNet18. Models are implemented in PyTorch and trained on one Tesla P100 GPU.

B. Experimental Settings

  • The most important hyperparameters for performance are r0 and T, searched over [0.1, 2.0] and [0.01, 1.0], respectively.Source: B. Experimental Settings · p. 14
  • The experiments fix h = 1 and R = 3, and apply InfoDrop in ResNet18 at the first convolutional layer and first residual block, or sometimes only the first layer.Source: B. Experimental Settings · p. 14
  • Hyperparameters are selected using validation results; implementation uses PyTorch and training uses a single NVIDIA Tesla P100 GPU.Source: B. Experimental Settings · p. 14

B.1.1. DATASET

The domain-generalization experiments use PACS, with training domains split into training and validation data and the full target domain reserved for testing. The appendix also specifies ResNet18 settings and describes visualizations of saliency and stylized images.

B.1.1. DATASET

  • PACS contains four domains and seven shared categories; the protocol splits source-domain images 9:1 for training and validation and tests on the entire target domain.Augmentation uses random crops to 80–100% of the original image size and random horizontal flips.Source: §B.1.1 · p. 14

B.1.2. PARAMETER SETUP

  • ResNet18 models train for 100 epochs with SGD; InfoDrop uses bandwidth h=1 and radius R=3, with source-specific r0 and T settings.For photo, art, or cartoon sources, r0=1.5; T is 0.03 for photo and 0.01 for art or cartoon. For sketch, r0=1.2 and T=1.0.Source: §B.1.2 · p. 14
  • Figure 6 places vanilla CNN saliency maps, InfoDrop saliency maps, and original images from left to right; Figure 7 shows stylized images, edge detections, and self-information distributions.Figure 7 arranges these in top, middle, and bottom rows, respectively.Source: Figure 6 · p. 15, Figure 7 · p. 16

B.2.1. DATASET

The few-shot experiments use mini-ImageNet and CUB, including a cross-domain mini-ImageNet-to-CUB setting. They use a Conv-4 backbone and evaluate results over repeated novel-class episodes.

B.2.1. DATASET

  • mini-ImageNet uses 64 training, 16 validation, and 20 novel classes; CUB uses 100 base, 50 validation, and 50 novel classes, with an additional mini-ImageNet-to-CUB setting.mini-ImageNet has 100 classes and 600 images per class; CUB has 200 classes and 11,788 images.Source: §B.2.1 · p. 17

B.2.2. PARAMETER SETUP

  • Few-shot experiments use Conv-4 and Adam, training from scratch for 60,000 5-way 5-shot episodes or 80,000 5-way 1-shot episodes; InfoDrop is applied in the first two convolutional layers.When augmentation is used, training adds 20,000 episodes; meta-testing averages 600 experiments.Source: §B.2.2 · p. 17
  • Baseline results from the Chen et al. (2019) reimplementation may be higher than results reported in the original papers.The appendix directs readers to Chen et al. (2019) for details.Source: §B.2.2 · p. 17

B.3.1. DATASET

The corruption experiments use Caltech-256 clean images and generated corruptions, with the highest severity level selected. The appendix specifies the ResNet18 training setup and limits comparisons to incremental effects.

B.3.1. DATASET

  • Caltech-256 supplies clean images, with 20% manually reserved for testing; corruptions cover 18 types at level-5 severity.Rescaling and random cropping are used for augmentation.Source: §B.3.1 · p. 17

B.3.2. PARAMETER SETUP

  • ResNet18 models train for 10 epochs, with InfoDrop applied before the residual blocks in the first convolutional layer.For adversarial training, the setup uses 20-run PGD with an l∞ norm of 1/255; the stated goal is incremental evaluation, not direct comparison against adversarial training.Source: §B.3.2 · p. 17

B.4.1. DATASET

Adversarial robustness is evaluated on Caltech-256 and CIFAR10 using dataset-specific protocols. The appendix also identifies settings for experiments examining InfoDrop's shape bias.

B.4.1. DATASET

  • Adversarial robustness experiments use Caltech-256 with a manually selected 20% test set and CIFAR10 following Zhang et al. (2019).Caltech-256 uses rescaling and random-crop augmentation.Source: §B.4.1 · pp. 17–18

B.4.2. PARAMETER SETUP

  • For Caltech-256, InfoDrop is applied to the first convolutional layer and first residual block; for CIFAR10, it is applied only to the first convolutional layer of ResNet18.Caltech-256 models train for 10 epochs, while CIFAR10 models train for 105 epochs; both setups use 20-run PGD attacks.Source: §B.4.2 · p. 18

B.5. Shape-bias of InfoDrop

  • Saliency-map and patch-shuffling experiments use PACS photo-domain images; style-transfer experiments use pretrained ResNet18 fine-tuned on content and style images.The style-transfer setup uses images from the AdaIN-style repository.Source: §B.5 · p. 18

Related papers

Loading 2008.04254v1…