PaperlayerView exact revision on arXiv

Source-linked AI summary

Robust and Generalizable Visual Representation Learning via Random Convolutions

Zhenlin Xu, Deyi Liu, Junlin Yang, Colin Raffel, Marc Niethammer

arXiv:2007.13003v3cs.CVcs.LG

TL;DR

Deep neural networks remain vulnerable to texture shifts, domain shifts, and small perturbations despite large-scale training. The paper introduces shape-preserving random convolutions as texture-randomizing augmentation, and reports stronger unseen-domain performance, including large gains on sketch benchmarks and benefits from robust pretraining.

  • Problem

    Deep neural networks often rely on local textures and lack robustness to domain shifts and small perturbations, motivating representations invariant to local texture.

  • Method

    RandConv uses multi-scale random convolutions to generate random-texture images with consistent shapes, either as training images or mixed with originals, with consistency regularization encouraging texture invariance.

  • Results

    RandConv consistently improves unseen-domain performance, outperforming state-of-the-art methods by a large margin on digit recognition, PACS Sketch, and ImageNet-Sketch.

  • Takeaways & Limitations

    A model pretrained with RandConv can transfer improved generalizability to downstream tasks on unseen domains.

  • Takeaways & Limitations

    Texture-invariant representations may hurt in-domain performance when local texture is useful, especially for fixed-domain fine-grained recognition.

Abstract

from arXiv · show

While successful for various computer vision tasks, deep neural networks have shown to be vulnerable to texture style shifts and small perturbations to which humans are robust. In this work, we show that the robustness of neural networks can be greatly improved through the use of random convolutions as data augmentation. Random convolutions are approximately shape-preserving and may distort local textures. Intuitively, randomized convolutions create an infinite number of new domains with similar global shapes but random local textures. Therefore, we explore using outputs of multi-scale random convolutions as new images or mixing them with the original images during training. When applying a network trained with our approach to unseen domains, our method consistently improves the performance on domain generalization benchmarks and is scalable to ImageNet. In particular, in the challenging scenario of generalizing to the sketch domain in PACS and to ImageNet-Sketch, our method outperforms state-of-art methods by a large margin. More interestingly, our method can benefit downstream tasks by providing a more robust pretrained visual representation.

1 INTRODUCTION

The paper targets robust visual representation learning from single-domain data by making models invariant to local texture while preserving global shape. It proposes RandConv and evaluates multi-scale, mixed, and consistency-based variants for unseen-domain generalization and transferable pretraining.

  • RandConv addresses robust visual representation learning from single-domain data, where limited work has previously been available.
  • The authors also test whether a more generalizable ImageNet-pretrained representation improves domain generalization on new downstream tasks.
  • RandConv uses multi-scale random convolutions to create random-texture images while maintaining global shapes.
  • The method trains with either RandConv outputs as images or mixtures of those outputs with originals, with consistency loss enforcing texture invariance.
  • The paper validates RandConv on synthetic and real-world benchmarks and ImageNet, including digit recognition, PACS Sketch, and ImageNet-Sketch.

2 RELATED WORK

Related work addresses domain generalization through feature fusion, meta-learning, adversarial training, domain randomization, augmentation, self-supervision, and shape-biased representation learning. Much prior domain-generalization work relies on multiple source domains or additional data and computation.

  • Domain generalization seeks representations that perform well when transferred to unseen domains.
  • Existing domain-generalization techniques include feature fusion, meta-learning, adversarial training, domain randomization, data augmentation, and self-supervised learning.
  • Most current domain-generalization work requires multi-source training, although collecting data from multiple sources can be difficult and expensive.
  • Domain randomization can synthesize diversity through randomized backgrounds, colors, lighting, and textures, while image stylization can generate new domains when simulation is unavailable.
  • Shape-biased representation methods counter neural networks’ reliance on superficial local features by using stylized data or penalizing correlations with superficial features.
  • Random mappings have been used for dimensionality reduction, adversarial robustness, reinforcement-learning generalization, and meaningful representations from fixed random weights.

3 RANDCONV: RANDOMIZE LOCAL TEXTURE AT DIFFERENT SCALES

RandConv inserts randomly weighted convolutional transformations during training to produce shape-consistent images with randomized local texture. Multi-scale filters, optional input mixing, and consistency regularization extend the augmentation across texture scales and encourage invariant predictions.

  • RandConv uses a non-fixed random convolution layer during training to generate images with random local texture and consistent shapes.
  • A random convolution output is g = I ∗ Θ, with spatial dimensions preserved by appropriate padding while the output channel count is Cout.
  • Random projections approximately preserve relative similarity between local patches, allowing sufficiently small filters to preserve larger-scale shapes.
  • Filter size determines the smallest shape preserved: 1x1 convolutions act as random color mappings, whereas larger filters perturb smaller shapes.
  • RCimg: Augmenting Images with Random Texture: RCimg uses randomized convolution outputs as new training images, yielding infinitely many random domains with consistent global shape and random texture.
  • RCmix: Mixing Variant: RCmix blends each original image with its RandConv output as αI + (1 − α)(I ∗ Θ), continuously interpolating between the training domain and randomized domains.
  • Multi-scale RandConv samples filter sizes from a pool and Gaussian convolution weights so shapes are preserved at varying scales.
  • Consistency Regularization: Consistency regularization uses KL divergence to encourage consistent predictions for different random-filter samples of the same augmented image.

4 EXPERIMENTS

Experiments evaluate RandConv in single-domain generalization across digit datasets, PACS, ImageNet-Sketch, and downstream PACS transfer. Across these settings, RandConv improves out-of-domain performance, especially for sketch-like domains, while reduced texture bias can slightly hurt Photo performance.

  • Experimental setup: Experiments use single-domain generalization across digit recognition, PACS, and ImageNet-Sketch, plus PACS transfer from ImageNet-pretrained representations.Training and validation data are drawn from one domain in the digit and transfer experiments; PACS follows pooled multi-source training without domain labels.
  • Digit recognition: 69.19 DG-Avg results from adding only 10% RandConv data, increasing average performance from 53.53 across MNIST-M, SVHN, SYNTH, and USPS.The experiment selected p = 0.5 for subsequent studies because it achieved the best DG-Avg.
  • Digit recognition: RandConv variants significantly outperform competing methods on digit domain generalization and achieve 91.62% accuracy on MNIST-C.RCmix performs marginally better than RCimg on both DG-Avg and MNIST-C.
  • PACS experiments: RandConv significantly improves PACS Sketch performance; RCmix improves over Deep-All by approximately 4% without consistency loss, while consistency loss improves Sketch but degrades the other domains.Sketch has no color and much less texture than the other PACS domains, making it the most challenging target.
  • ImageNet-Sketch: RandConv outperforms PAR by 5% on ImageNet-Sketch and matches a Stylized-ImageNet-trained model despite training the RandConv model from scratch.PAR uses a stronger baseline, while PAR and SIN fine-tune baseline models.
  • Robust pretrained representations: 73.03% is the best average PACS domain-generalization accuracy, improving the initial Deep-All baseline by more than 6%.Using RandConv-trained ImageNet models as initialization improves Sketch and Art, while Photo performance drops slightly as expected from reduced texture bias.

5 CONCLUSION AND DISCUSSION

RandConv improves robustness by randomizing local texture while preserving global shape, including for unseen domains and downstream tasks. Its benefits are strongest when shape bias is useful, while texture invariance can hurt fixed-domain fine-grained recognition.

  • Conclusion: RandConv randomizes local image texture, emphasizes global shape information, and outperforms state-of-the-art approaches on several domain-generalization benchmarks.The reported benchmarks include digit recognition, PACS Sketch, and ImageNet-Sketch.
  • Conclusion: Finetuning a RandConv-pretrained model on PACS transfers robustness to a new downstream task and achieves new state-of-the-art performance on PACS Sketch.
  • Discussion: RandConv is most useful when a shape-biased model is helpful, whereas texture-invariant representations may hurt fixed-domain fine-grained recognition.
  • Discussion: Future work includes disentangling shape and texture features and using those representations in an explainable way.

A SHAPES AND TEXTURE IN IMAGES

The paper defines preserved image structures as primitive shapes: spatial pixel clusters sharing similar local texture. Random convolutions preserve larger semantic shapes while potentially distorting smaller shapes treated as local texture.

  • Shapes and texture: Primitive shapes are spatial clusters of pixels with similar local texture that remain preserved by a random convolution layer.
  • Shapes and texture: Objects usually comprise multiple primitive shapes, such as a car’s wheels, body frames, and windshields.
  • Shapes and texture: Texture and shape are not strict opposites because a larger shape can contain smaller shapes that function as its texture.
  • Shape-preserving property: A shape-preserving transformation maintains the relative closeness of patches within one shape compared with patches from neighboring shapes.

B RANDOM CONVOLUTION IS SHAPE-PRESERVING AS RANDOM LINEAR PROJECTION IS DISTANCE PRESERVING

The paper models convolution as a local linear projection and argues that Gaussian random projections approximately preserve relative distances, supporting shape preservation. Theory and image simulations quantify this behavior.

  • Random convolution: A convolution layer maps each vectorized local image patch p(x, y) from R^d to an output feature g(x, y) in R^Cout through a linear projection U.Here d = h × w × C_in, and U expresses the convolution filters.
  • Shape preservation: Random Gaussian convolution filters approximately preserve relative similarity between local patches, so sufficiently large shapes tend to remain recognizable in the output feature map.
  • Theoretical analysis: For N points, Theorem 1 bounds pairwise distance rescaling after a random projection with probability 1 − ϵ; smaller N and larger m yield tighter bounds.
  • Empirical verification: With m = 3, N = 1,000, σ = 1, and ϵ = 0.1, 80% of real-image patch pairs had rescaling ratios in [0.56, 2.87], tighter than the theoretical bound [0.01, 5.8].The simulation used 1,000 PACS images and 3 × 3 patches, giving d = 27.

C EXPERIMENTAL DETAILS

The experiments use task-specific training procedures for digit recognition, PACS domain generalization, and ImageNet model training, with standard convolutional-network optimization setups.

  • Digits Recognition: Digit recognition uses two Conv5×5-ReLU-MaxPool2×2 blocks followed by three fully connected layers and trains for 10,000 iterations.Training uses batch size 32 and Adam with initial learning rate 0.0001.
  • PACS: PACS experiments use official splits, pretrained AlexNet weights, mixed samples from three source domains, and 50,000 finetuning iterations.Validation uses source-domain data during training.
  • ImageNet: ImageNet experiments train AlexNet from scratch for 90 epochs with batch size 256 and SGD.The initial learning rate is 0.01 with momentum 0.9 and weight decay 0.0001.

D MORE EXPERIMENTS WITH RESNET-18

These experiments test RandConv with ResNet-18 on ImageNet-Sketch, ImageNet-R, and PACS, finding improved out-of-domain performance and stronger downstream finetuning.

  • 10.5% accuracy improvement over the ResNet-18 baseline is reported on ImageNet-Sketch with RandConv.
  • RandConv-pretrained ResNet-18 improves PACS finetuning performance for both DeepAll and RandConv.The PACS results are reported in Table 7, which compares finetuning methods using RandConv-pretrained models.
  • RandConv improves generalization on ImageNet-R and reduces the performance gap between ImageNet-200 and ImageNet-R.ImageNet-200 contains the original ImageNet data restricted to ImageNet-R’s 200 classes.
  • JiGen improves over its own baseline by 1.5%, while using extra augmentation and a different data split from this approach.

E HYPERPARAMETER SELECTIONS AND ABLATION STUDIES ON DIGITS RECOGNITION BENCHMARKS

The digits experiments examine hyperparameter p, the multi-scale design, and consistency-loss weight λ through ablation studies across multiple RandConv variants.

  • Hyperparameter p is evaluated for RCimg1 on digits-recognition benchmarks using DG-Avg across MNIST-M, SVHN, SYNTH, and USPS.Table 8 reports the corresponding ablation results.
  • The multi-scale design is ablated for RCmix and RCimg with p=0.5 on digits-recognition benchmarks.These experiments correspond to Table 9 and assess the effect of multi-scale RandConv.
  • Consistency-loss weight λ is studied for RCmix1-7 and RCimg1-7 with p=0.5 on digits-recognition benchmarks.Table 10 reports DG-Avg across MNIST-M, SVHN, SYNTH, and USPS.
  • DG-Avg summarizes average performance across MNIST-M, SVHN, SYNTH, and USPS for the digits-recognition ablations.

F MORE EXAMPLES OF RA NDCONV DATA AUGMENTATION

Additional examples illustrate how RandConv filter size affects shape retention and how mixing coefficients interpolate between original and randomly convolved images.

  • Different filter sizes retain shapes at different scales, with smaller filters better maintaining finer shapes.Figure 6 shows the input followed by convolution results from random filters of different sizes k.
  • When α = 1, RCmix7 returns the original image, while α = 0 uses the random-convolution output as the augmented image.Intermediate coefficients continuously interpolate between the training domain and a randomly sampled domain.
Loading 2007.13003v3…