Learning Robust Representations by Projecting Superficial Statistics Out

Haohan Wang; Zexue He; Zachary C. Lipton; Eric P. Xing

Source-linked AI summary

Learning Robust Representations by Projecting Superficial Statistics Out

Haohan Wang, Zexue He, Zachary C. Lipton, Eric P. Xing

arXiv:1903.06256v1Mar 2, 2019cs.CVcs.LG

TL;DR

The paper addresses the fragility of deep classifiers that exploit superficial statistics under distribution shift, especially when target domains and domain identifiers are unavailable. It uses GLCM-derived texture representations with reverse-gradient or orthogonal-projection techniques to remove such information, and reports comparable or better performance than methods requiring target-distribution samples.

Problem
Deep neural networks can rely on superficial training statistics and fail under distribution shift, motivating domain-generalizing classifiers that work on unseen domains without domain identifiers during training.
Method
The paper uses a neural gray-level co-occurrence matrix to capture texture and removes that information through reverse-gradient training or projection onto a subspace orthogonal to the texture representation.
Results
The method achieves comparable or better performance than other domain-generalization methods that explicitly require target-distribution samples for training.
Takeaways & Limitations
Discarding representations associated with known superficial visual statistics can support out-of-sample performance across unseen domains.
Takeaways & Limitations
NGLCM cannot be completely free of image semantics, so projecting its representation out can slightly reduce performance on standard MNIST.

Abstract

from arXiv · show

Despite impressive performance as evaluated on i.i.d. holdout data, deep neural networks depend heavily on superficial statistics of the training data and are liable to break under distribution shift. For example, subtle changes to the background or texture of an image can break a seemingly powerful classifier. Building on previous work on domain generalization, we hope to produce a classifier that will generalize to previously unseen domains, even when domain identifiers are not available during training. This setting is challenging because the model may extract many distribution-specific (superficial) signals together with distribution-agnostic (semantic) signals. To overcome this challenge, we incorporate the gray-level co-occurrence matrix (GLCM) to extract patterns that our prior knowledge suggests are superficial: they are sensitive to the texture but unable to capture the gestalt of an image. Then we introduce two techniques for improving our networks' out-of-sample performance. The first method is built on the reverse gradient method that pushes our model to learn representations from which the GLCM representation is not predictable. The second method is built on the independence introduced by projecting the model's representation onto the subspace orthogonal to GLCM representation's. We test our method on the battery of standard domain generalization data sets and, interestingly, achieve comparable or better performance as compared to other domain generalization methods that explicitly require samples from the target distribution for training.

1 INTRODUCTION

The paper targets classifiers that remain robust when superficial visual cues change across domains. It proposes extracting textural information and discarding it so models rely more on semantic information.

Distribution shift can expose classifiers that rely on correlations between labels and superficial image properties such as backgrounds.The paper illustrates training and validation backgrounds correlated with sentiment labels, while test backgrounds are independent.
Domain adaptation has established methods under restrictive shift assumptions, but domain generalization excludes target-domain samples during training.
The paper assumes that reducing dependence on known superficial statistics can encourage reliance on semantically meaningful differences.Its visual focus is high-frequency textural information.
The proposed neural gray-level co-occurrence matrix captures textural information from images without modeling the lower-frequency semantic information of interest.
The proposed architecture-agnostic, parameter-free method is designed to discard superficial information from the model representation.
The paper introduces two synthetic domain-adaptation/domain-generalization datasets where domain-specific information is correlated with semantic information.Figure 1 provides a toy example of this correlation structure.

2 RELATED WORK IN DOMAIN ADAPTATION AND DOMAIN GENERALIZATION

The related work situates domain generalization as a target-free variant of domain adaptation and surveys representation alignment, domain-specific ensembles, and metalearning approaches.

Domain generalization differs from domain adaptation because target-domain samples are unavailable during training.Datasets may combine heterogeneous sources without labeling those sources.
Deep domain-generalization methods commonly reduce inter-domain representation differences through adversarial techniques or fuse representations from one model per domain.
Metalearning and zero-shot domain adaptation are also identified as related approaches.

3 METHOD

The method introduces a differentiable neural GLCM to isolate texture-like information, then reduces its influence on classification through adversarial prediction or projection-based transformations. The resulting classifier is trained to use raw-image representations that are less explainable by superficial textural representations.

3.1 NEURAL GRAY-LEVEL CO-OCCURRENCE MATRIX FOR SUPERFICIAL INFORMATION: The GLCM counts pixel-value pairs under a chosen direction, motivating NGLCM’s representation of texture rather than image gestalt.The paper describes horizontal, 45°, 90°, and 135° directions as common choices.
3.1 NEURAL GRAY-LEVEL CO-OCCURRENCE MATRIX FOR SUPERFICIAL INFORMATION: NGLCM is a differentiable neural building block designed to extract textural information without modeling lower-frequency semantic information.It resembles a gray-level co-occurrence matrix while providing subgradients for backpropagation.
3.1 NEURAL GRAY-LEVEL CO-OCCURRENCE MATRIX FOR SUPERFICIAL INFORMATION: NGLCM replaces nondifferentiable counting with tunable thresholding parameters and differentiable matrix operations that can be optimized through backpropagation.Under an additional uniqueness constraint on φ, the construction is equivalent to conventional GLCM up to permutation and scale; the authors drop that constraint in practice.
3.2 HEX: The classifier concatenates raw-data and textural representations, then seeks predictions from the raw representation that are independent of the superficial representation.The final layer receives h(X; θ) and g(X; φ) as concatenated inputs before the independence-oriented transformation.
3.2 HEX: ADV uses reverse gradients to keep the raw representation predictive of semantic labels while making the textural representation unpredictable from it.An MLP variant that predicts g(X; φ) from h(X; θ) is called ADVE.
3.2 HEX: A projection-based alternative transforms representation A using a projection matrix constructed from representation B to make A least explainable by B.The paper refers to the residual-maker-matrix formulation and uses the transformed representation for testing.

4 EXPERIMENTS

The experiments evaluate HEX and related methods on synthetic and standard domain-generalization datasets, including settings where superficial signals correlate with semantic labels. Across these tests, HEX is comparatively stable under stronger shifts and performs competitively with established methods.

Experimental setup: The experiments assess HEX’s resilience to dataset shift using two synthetic datasets and two standard domain-generalization datasets.The synthetic evaluations introduce shifts at semantic and raw-feature levels; all models are trained with ADAM.
Synthetic datasets: The facial-expression dataset correlates seven backgrounds with seven sentiment labels in training and validation, then makes backgrounds independent of sentiment at test time.The split is 50% training, 30% validation, and 20% testing, with correlation level ρ controlling the shift.
Facial expression classification: As correlation ρ increases, most methods’ testing performance decreases, whereas ADV and HEX remain relatively stable across ten correlation settings.Testing accuracy and standard deviation are averaged over five repeated experiments.
MNIST surface statistics: On MNIST with attached patterns, M performs best when patterns are independent, while N and HEX perform best when patterns are dependent.In the most challenging setting, with random-kerneled testing and dependent pattern attachment, HEX shows a clear advantage and is relatively stable overall.
Standard domain-generalization datasets: On MNIST-Rotation, HEX is inferior to previous methods in only one case and leads average performance overall.The comparison includes CAE, MTAE, CCSA, DANN, Fusion, LabelGrad, and CrossGrad.
Standard domain-generalization datasets: On PACS, HEX is only slightly below Fusion overall despite using no explicit training-domain identification and a substantially smaller model.HEX performs particularly strongly on Art painting and Cartoon, while Fusion performs well on Photo and Sketch.

5 DISCUSSION AND CONCLUSION

The paper presents NGLCM and HEX as components for removing textural information and notes limitations involving semantic leakage, training instability, and optimization heuristics.

NGLCM extracts textural information, while HEX projects that information out to encourage reliance on semantic information.
NGLCM cannot be completely free of semantic information, so projecting its representation out can slightly reduce performance on standard MNIST.
Simultaneously training all model parameters can produce a trivial solution in which FG learns garbage information and HEX degenerates to the baseline model.
The authors use sequential optimization and fixed weights as training heuristics, but generally do not report heuristic-training results except for PACS.
HEX training can fluctuate dramatically, although selecting the model with highest validation accuracy generally produces better results than competing methods.

A1 REASONS TO CHOOSE GLCM

The authors select GLCM because experiments indicate that it captures textural patterns better than alternatives while predicting semantic tasks less effectively.

The authors compare SURF, LBP, and GLCM across datasets containing semantic and superficial information to identify a suitable texture extractor.
GLCM outperforms the other methods in most cases when classifying textural patterns.
GLCM predicts semantic tasks least well among the compared methods, supporting its intended separation of texture from semantics.

A2.1 MATHEMATICAL RATIONALE

HEX transforms the model representation by regressing out the portion explainable by the textural representation, rather than relying on direct subtraction.

The method seeks a transformation of FA that is least explainable by FG.
Direct subtraction is considered problematic because FA − FG can remain correlated with FG.
Because FA and FG share a space and the remaining network operation is linear argmax, the authors use linear operations for regression.
The regression is formulated column-wise by considering each column of FA as a separate target.
A closed-form solution is available when the minibatch has more rows in FG than FG has columns, corresponding to more samples than classes.
Repeating the residual calculation for every column of FA yields the matrix form identified as Equation 4.

A2.2 WHEN F T

The closed-form HEX derivation depends on minibatch size and matrix invertibility; larger minibatches provide a simple solution, while the general alternative uses heteroscedastic regression but is slower and noise-sensitive.

Equation 4 requires a minibatch size greater than the number of classes because the relevant matrix must be invertible.
Using a minibatch larger than the number of classes guarantees a HEX solution and is presented as realistic for typical applications.
The authors also introduce a method intended to remain applicable independently of minibatch size.
Adding λI to F_G^T F_G guarantees invertibility, but treating λ as a tunable hyperparameter conflicts with the preference for hyperparameter-free operation.
Heteroscedastic regression can estimate λ through maximum likelihood, completing the hyperparameter-free approach when F_G^T F_G is not invertible.
The maximum-likelihood procedure is very slow and sensitive to noise, so the authors recommend choosing a larger minibatch size in practice.

A3 EXTRA EXPERIMENT RESULTS

The extra experiments examine superficial visual patterns and optimization behavior across Office, facial-expression, and synthetic MNIST settings. Office images reveal domain-specific background differences, while HEX and related ablations converge faster than baseline methods.

Office data set: Office images from DSLR and Webcam share similar backgrounds, whereas Amazon images have a distinctly different background.The visualization covers 12 of 31 labels, with the remaining labels described as showing a similar pattern.
Facial expression classification: HEX and related ablation methods converge significantly faster than baseline methods in facial-expression classification.The comparison uses testing-accuracy curves shown in Figure A2.
Synthetic MNIST: Synthetic MNIST images are augmented with radial, random, and original-image surface regularities using Fourier filtering patterns.Figure A3 illustrates examples of images with different kernelled patterns and identifies the leftmost image as the kernel.