PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Robust Global Representations by Penalizing Local Predictive Power

Haohan Wang, Songwei Ge, Eric P. Xing, Zachary C. Lipton

arXiv:1905.13549v2cs.CV

TL;DR

The paper addresses the problem that convolutional classifiers can rely on locally predictive signals that may fail under distribution shift. It introduces Patch-wise Adversarial Regularization, which penalizes early local representations while preserving final classification, and reports improved out-of-domain performance across synthetic and benchmark tasks, including ImageNet-Sketch. Its main scope boundary is that local patterns can be genuinely predictive in-domain, where the method may not help or may reduce accuracy.

  • Problem

    Convolutional classifiers may rely on locally predictive but semantically unrelated signals, creating a need for robust performance under out-of-domain shifts.

  • Method

    Patch-wise Adversarial Regularization penalizes the predictive power of local representations in early layers by training side classifiers adversarially against the network’s final classifier.

  • Results

    Across synthetic and real domain-adaptation and domain-generalization datasets, the methods favor improved performance, especially when domain information is unknown, and also perform favorably on ImageNet-Sketch.

  • Takeaways & Limitations

    Suppressing local predictive signals can encourage global concepts and improve robustness to distribution shifts without relying on domain information.

  • Takeaways & Limitations

    PAR may not help or may hurt in-domain accuracy when local patterns are truly predictive, and no PAR variant consistently outperforms vanilla PAR.

Abstract

from arXiv · show

Despite their renowned predictive power on i.i.d. data, convolutional neural networks are known to rely more on high-frequency patterns that humans deem superficial than on low-frequency patterns that agree better with intuitions about what constitutes category membership. This paper proposes a method for training robust convolutional networks by penalizing the predictive power of the local representations learned by earlier layers. Intuitively, our networks are forced to discard predictive signals such as color and texture that can be gleaned from local receptive fields and to rely instead on the global structures of the image. Across a battery of synthetic and benchmark domain adaptation tasks, our method confers improved generalization out of the domain. Also, to evaluate cross-domain transfer, we introduce ImageNet-Sketch, a new dataset consisting of sketch-like images, that matches the ImageNet classification validation set in categories and scale.

1 Introduction

The paper targets out-of-domain failures caused by classifiers relying on locally predictive but semantically unrelated image signals. It introduces Patch-wise Adversarial Regularization to suppress local predictive power and encourage global concepts, with experiments reporting stronger performance across diverse shifts.

  • Motivation: Backgrounds can predict object categories in training images even when they are semantically unrelated to the objects themselves.Sea turtles are often photographed in the sea, whereas tortoises are often photographed on land.
  • Motivation: Correlated artifacts may change out-of-sample, motivating classifiers that remain accurate when familiar objects appear in unfamiliar contexts.The supplied passage gives indoor cats and outdoor cats as an example of this concern.
  • Problem setting: Domain adaptation studies robust classification under distribution shifts, but general domain adaptation is impossible without restrictions relating training and test distributions.The paper notes that progress remains possible under assumptions such as covariate shift or label shift.
  • Motivation: The authors posit that suppressing locally predictive patches and using concepts derived across regions may better mimic human robustness in visual recognition.The proposed motivation distinguishes small local features from larger-scale image structure.
  • Contribution: Patch-wise Adversarial Regularization penalizes predictive power in early local representations while preserving correct final-layer classification.It applies patch-wise classifiers at spatial locations and trains the network to fool them through reverse gradients.
  • Results: Across synthetic and real datasets, the method outperforms competing methods, especially when domain information is unavailable, and its performance is endorsed on a new ImageNet-scale sketch benchmark.ImageNet-Sketch matches ImageNet validation categories and scale while using sketch-like images.

2 Related Work

Related work spans principled domain adaptation assumptions, heuristic deep-learning approaches, and domain generalization methods that seek robustness without target-domain data. The paper extends prior work toward image classifiers robust to natural shifts without using domain information.

  • Domain adaptation: Classical domain adaptation work studies settings such as covariate shift, label shift, and other assumptions relating source and target distributions.The cited literature includes methods where p(y|x) or p(x|y) remains unchanged.
  • Deep-learning approaches: Heuristic deep-learning methods can improve benchmarks even when traditional domain-adaptation assumptions, such as shared support, break down.These methods lack theoretical backing according to the passage.
  • Deep-learning approaches: Modern domain-adaptation methods broadly fine-tune on target data or enforce domain invariance through adversarial learning.The passage identifies these as two high-level subtypes.
  • Domain generalization: Domain generalization seeks robustness to distribution shifts without access to target-distribution data, including unlabeled data.Some formulations use domain identifiers from known domains, while related work also studies zero-shot domain adaptation.
  • Positioning: The paper extends a method using domain-sensitive components and studies robustness to natural shifts without domain information during training or deployment.This places the paper in the domain-generalization setting described in the related work.

3 Method

PAR adds local side classifiers to early convolutional representations and adversarially suppresses their predictive power, encouraging higher layers to use information aggregated across regions. The method supports architectural variants and a staged training heuristic.

  • Base formulation: The network is written as f(g(·; δ); θ), where g produces bottom-layer representations and δ and θ are learned parameters.The standard training objective uses a loss such as cross-entropy for classification.
  • Base formulation: For each input, g produces a c × m′ × n′ activation tensor whose spatial locations correspond to local receptive-field representations.The exact output dimensions depend on image, padding, and stride settings.
  • Patch-wise classifier: A shared classifier h maps each c-length local vector to a class prediction, yielding m′ × n′ predictions from individual image patches.The same parameters are reused across spatial locations, enabling local patterns to be detected regardless of their position.
  • Adversarial regularization: PAR trains h to predict labels from local features while training g to shift away from locally predictive representations.The joint optimization aims to preserve final prediction accuracy while reducing reliance on local patterns.
  • Adversarial regularization: The regularization strength is controlled by λ, and the local loss is divided by m′n′ so its scale matches the standard objective.λ is a tuning hyperparameter.
  • Implementation: PAR uses a 1×1 convolution to implement the shared patch-wise fully connected classifier efficiently across all spatial locations.The operation has c input channels and k output channels, where k is the number of classes.
  • Extensions: The variants enlarge classifier capacity, local context, or representational level through PARM, PARB, and PARH.PARM uses a three-layer ReLU MLP; PARB uses 3×3 convolutions; PARH regularizes a higher convolutional layer.
  • Training heuristics: A staged heuristic first trains the model conventionally and then continues training with PAR regularization, including from pretrained models.This schedule is described as particularly useful for architectures such as AlexNet or ResNet.

4 Experiments

Experiments across perturbed MNIST and CIFAR10, PACS, and ImageNet-Sketch evaluate PAR for domain generalization and adaptation. PAR and its variants generally improve out-of-domain performance, with especially notable gains on sketch-like or otherwise locally perturbed domains.

  • 4.1 MNIST with Perturbation: PAR outperformed competing methods in most perturbed MNIST settings, although its variants behaved differently depending on the test pattern.PARM performed best for the “original” and “radial” settings but nearly worst for the “random” setting.
  • 4.2 CIFAR with Perturbation: PAR and its variants achieved the best performance across all four perturbed CIFAR10 test sets, despite DANN using unlabeled test data during training.PAR was best on greyscale and radial-kernel settings, while PARM was best on negative-color and random-kernel settings.
  • 4.3 PACS: PAR improved the PACS AlexNet baseline by 2% after fine-tuning on training-domain data.The PACS comparison separates methods that use domain identifications from those that do not.
  • 4.3 PACS: PACS results showed relatively large improvements on Sketch, the only colorless domain among its four domains.The authors connect this pattern to models’ tendency to exploit local color information when other domains are used for testing.
  • 4.4.1 The ImageNet-Sketch Data: ImageNet-Sketch was constructed as a large-scale out-of-domain benchmark with ImageNet-like categories and scale, using black-and-white sketch images.Sample classes include magpie, box turtle, goldfish, golden retriever, parachute, bookshop, acoustic guitar, racer, and giant panda.
  • 4.4.2 Experiment Results: On ImageNet-Sketch, PAR and AlexNet shared 5025 correct predictions, while AlexNet alone correctly predicted 1098 images and PAR alone correctly predicted 1617.The differing correct-prediction sets indicate that the aggregate improvement masks substantial changes in which examples are classified correctly.

5 Conclusion

The paper introduces Patch-wise Adversarial Regularization to encourage global concepts by penalizing local-patch predictive power, and reports improved performance across domain adaptation and generalization experiments. It also introduces ImageNet-Sketch as a large-scale benchmark for out-of-domain evaluation, while acknowledging important limitations and variant-selection uncertainty.

  • Across domain adaptation and domain generalization tasks, the proposed methods generally performed best when domain information was unavailable.
  • ImageNet-Sketch matches ImageNet’s validation-set categories and scale while containing sketch-like images for evaluating out-of-domain generalization.
  • The method may fail to improve or may reduce in-domain accuracy when local patterns are genuinely predictive of labels.
  • No PAR variant consistently outperformed vanilla PAR, so variant selection remains dependent on the application scenario.

A Other Hyperparameter Choices for MNIST experiment

The MNIST sensitivity analysis varies PAR’s λ across five values and reports that relatively small λ values are preferred. However, later ImageNet-Sketch experiments show that λ = 1 can be too strong unless paired with a much smaller learning rate.

  • λ was varied across {0.01, 0.1, 1, 10, 100} to assess PAR’s sensitivity in the MNIST experiment.
  • PAR appeared to prefer relatively smaller λ values in the MNIST experiment, although λ = 1 was used in the main manuscript.
  • λ = 1 was too strong for ImageNet-Sketch unless the learning rate was set much smaller.
  • When training performance deteriorates, the authors recommend reducing λ or the learning rate.

B Cifar10 discussion

The CIFAR10 analysis evaluates perturbed color and texture conditions and examines how PAR training and architectural choices affect robustness. The reported figures describe local-pattern classifier behavior, training dynamics, filter-size comparisons, and adversarial-loss decays.

  • CIFAR10 examples were evaluated under perturbed color and texture conditions, including greyscale, negative color, random kernel, and radial kernel changes.
  • The patch-wise classifier achieved roughly 20% in-domain accuracy before regularization, versus 12% on texture-altered and 5% on color-altered out-of-domain data.
  • After adversarial training was introduced at epoch 250, perturbed-dataset performance increased while the model was coerced to forget local-pattern information.
  • Different convolutional filter sizes produced generally similar test-dataset performance, except for RandomKernel.
  • Using smaller decay, corresponding to larger λ, led to unstable performance when adversarial losses from four layers were weighted by 1, λ, λ2, and λ3.

C More results of ImageNet-Sketch

The ImageNet-Sketch analysis applies explicit comparison rules to identify informative disagreements between PAR and AlexNet. It reports additional cases where PAR was correct and AlexNet was wrong, attributing these differences to AlexNet’s focus on local patterns.

  • The analysis selects samples that one model predicts correctly and the other predicts incorrectly, while excluding patterns consistent with random or reciprocal class confusions.
  • The ImageNet-Sketch analysis therefore uses disagreement cases to examine whether local-pattern reliance explains model-specific errors.
  • Table 5 presents additional samples correctly classified by PAR but misclassified by AlexNet, which the authors conjecture reflects AlexNet’s focus on local patterns.
Loading 1905.13549v2…