PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Debiased Representation via Disentangled Feature Augmentation

Jungsoo Lee, Eungyeup Kim, Juyoung Lee, Jihyeon Lee, Jaegul Choo

arXiv:2107.01372v2cs.LG

TL;DR

Image classifiers can exploit peripheral bias attributes, while scarce bias-conflicting samples limit debiasing. The paper learns disentangled intrinsic and bias representations, swaps latent features to synthesize diverse conflicting samples, and reports state-of-the-art performance across synthetic and real-world datasets. It concludes that this augmentation strategy improves debiasing, while complete disentanglement remains challenging.

  • Problem

    Bias-conflicting samples are scarce in biased datasets, limiting debiasing and generalization when models rely on peripheral attributes correlated with labels.

  • Method

    The method learns disentangled intrinsic and bias representations with separate encoders, then swaps latent features to synthesize diversified bias-conflicting samples during training.

  • Results

    The approach achieves state-of-the-art debiasing performance against existing baselines on two synthetic datasets and one real-world dataset.

  • Takeaways & Limitations

    The experiments support increasing the diversity of bias-conflicting samples beyond the given training set as a route to learning debiased representations.

  • Takeaways & Limitations

    Completely disentangled representations remain challenging, especially with scarce conflicting samples and correlated complex attributes such as age and gender.

Abstract

from arXiv · show

Image classification models tend to make decisions based on peripheral attributes of data items that have strong correlation with a target variable (i.e., dataset bias). These biased models suffer from the poor generalization capability when evaluated on unbiased datasets. Existing approaches for debiasing often identify and emphasize those samples with no such correlation (i.e., bias-conflicting) without defining the bias type in advance. However, such bias-conflicting samples are significantly scarce in biased datasets, limiting the debiasing capability of these approaches. This paper first presents an empirical analysis revealing that training with "diverse" bias-conflicting samples beyond a given training set is crucial for debiasing as well as the generalization capability. Based on this observation, we propose a novel feature-level data augmentation technique in order to synthesize diverse bias-conflicting samples. To this end, our method learns the disentangled representation of (1) the intrinsic attributes (i.e., those inherently defining a certain class) and (2) bias attributes (i.e., peripheral attributes causing the bias), from a large number of bias-aligned samples, the bias attributes of which have strong correlation with the target variable. Using the disentangled representation, we synthesize bias-conflicting samples that contain the diverse intrinsic attributes of bias-aligned samples by swapping their latent features. By utilizing these diversified bias-conflicting features during the training, our approach achieves superior classification accuracy and debiasing results against the existing baselines on synthetic and real-world datasets.

1 Introduction

Dataset bias leads models to rely on peripheral attributes correlated with labels rather than intrinsic class attributes. Because bias-conflicting samples are scarce, the paper argues for diversifying them through disentangled feature augmentation.

  • Problem: Dataset bias causes models to learn peripheral attributes correlated with labels instead of intrinsic attributes defining a class.For example, bird images may correlate with blue-sky backgrounds while wings remain intrinsic attributes.
  • Existing limitations: Predefining bias types limits debiasing across other biases and requires costly manual identification.Existing approaches may be tailored to color or texture, restricting their applicability.
  • Motivation: Diverse bias-conflicting samples beyond the training set are crucial for learning debiased representations.The paper defines diversity as different valid realizations of intrinsic attributes within a class.
  • Approach: The method learns disentangled intrinsic and bias representations, swaps latent features, and schedules augmentation to synthesize diversified bias-conflicting features.Swapped features combine attributes from different images, reducing their correlation.
  • Results: The approach achieves state-of-the-art performance on Colored MNIST, Corrupted CIFAR-10, and Biased FFHQ against existing baselines.The evaluated datasets include two synthetic benchmarks and one real-world dataset.

2 Related Work

Prior debiasing methods either predefine bias types or exploit shortcut-learning assumptions without doing so. Latent-space augmentation offers an alternative, but bias-label correlation complicates generating suitable debiased features.

  • Predefined-bias methods: Methods that predefine bias types lack general applicability when the relevant biases are difficult to recognize.Examples include networks designed around color or texture biases.
  • Bias-type-agnostic methods: Other approaches avoid predefined bias types by emphasizing bias-conflicting samples or identifying shortcuts through early learning or latent gradients.These methods include generalized cross-entropy reweighting and high-gradient latent-vector assumptions.
  • Image-level augmentation: Image-level augmentation methods remain limited when they target a predefined texture bias.The cited approaches transfer styles or remove shortcut cues through image-to-image translation.
  • Latent-space augmentation: Latent-space augmentation can improve generalization, but strong bias-attribute and label correlation hinders obtaining debiased features.Prior methods perturb high-gradient latent vectors or mix styles across source domains.
  • Paper positioning: The paper addresses these limitations by diversifying bias-conflicting samples rather than relying only on scarce existing examples.Its contribution is positioned as augmentation-based debiasing without predefined bias types.

3 Importance of Diversity in Debiasing

Experiments on Colored MNIST and Corrupted CIFAR-10 show that the diversity of bias-conflicting samples matters more than their sampling ratio for debiasing. Diverse samples improve unbiased-test accuracy, even when sampled sparingly.

  • Datasets: The toy experiments use Colored MNIST and Corrupted CIFAR-10, each with four bias-conflicting-data ratios or correlation settings.Colored MNIST varies the bias-conflicting ratio across 0.5%, 1%, 2%, and 5%; Corrupted CIFAR-10 follows the same ratios.
  • Experimental settings: The study compares oversampling and low-ratio sampling using bias-conflicting pools with either 5% or 1% diversity.Oversampling uses 128 of 256 batch samples, while low-ratio sampling uses 2 of 256.
  • Findings: Diverse bias-conflicting samples outperform less-diverse samples on unbiased test accuracy, including when fewer diverse samples are sampled.The experiments compare four combinations of sampling ratio and diversity pool.
  • Interpretation: Diversity is more important than sampling ratio because limited diversity can cause overfitting to the available bias-conflicting examples.Missing intrinsic visual facets, such as scribbled digit shapes, may prevent learning generalized class attributes.

4 Debiasing via disentangled feature augmentation

The method disentangles intrinsic and bias attributes, then swaps their latent features to synthesize diverse bias-conflicting samples. Feature augmentation is scheduled after disentanglement develops, because earlier swapping can introduce noisy features.

  • Learning disentangled representation: Two encoder-classifier pairs separately learn latent representations of intrinsic attributes and bias attributes from input images.The intrinsic pathway uses (Ei, Ci), while the bias pathway uses (Eb, Cb).
  • Learning disentangled representation: Re-weighting emphasizes bias-conflicting examples when training the intrinsic encoder and classifier.The weighting function assigns higher values to bias-conflicting samples, increasing their contribution to intrinsic-feature learning.
  • Feature swapping for augmentation: The swap loss trains both networks on augmented features using the original label for intrinsic attributes and the permuted bias-attribute label for bias features.The total objective combines disentanglement and swapping losses, with λswap controlling augmentation weight.
  • Feature swapping for augmentation: Randomly permuting bias features across samples forms zswap = [zi; ˜zb], reducing intrinsic-bias correlation and creating diverse augmented bias-conflicting features.The intrinsic and bias features come from different images, while diversity is inherited from the many bias-aligned samples.
  • Scheduling the feature augmentation: Feature augmentation begins only after a scheduled iteration because early latent representations are insufficiently disentangled and can act as noisy samples.The approach uses an augmentation iteration threshold before enabling feature swapping.

5 Experiment

The experiments evaluate debiasing across synthetic and real-world datasets, then analyze augmentation components and disentangled representations. Results show strong test performance, with latent projections and reconstructions supporting separation of intrinsic and bias attributes.

  • Experiment details: Baselines include vanilla training, HEX, EnD, ReBias, and LfF, differing in whether they use predefined bias types or explicit bias labels.LfF requires no prior bias knowledge, whereas HEX and ReBias presume texture and EnD uses bias labels.
  • Experiment details: The evaluation uses Colored MNIST, Corrupted CIFAR-10, and Biased FFHQ, varying bias-conflicting ratios across datasets.Colored MNIST and Corrupted CIFAR-10 use unbiased test sets, while BFFHQ uses a bias-conflicting test set.
  • Quantitative evaluation: Our approach achieves state-of-the-art accuracy across all datasets against baselines without predefined bias types.The comparison uses image classification accuracy on the specified unbiased or bias-conflicting test sets.
  • Quantitative evaluation: The method outperforms HEX, ReBias, and EnD on the real-world dataset and performs comparably to ReBias on Colored MNIST without a predefined bias type.ReBias achieves the best Colored MNIST accuracy, while the proposed method achieves the second best there.
  • Analysis: Scheduled feature augmentation gives the best classification accuracy across datasets, supporting the importance of disentangled representation learning, augmentation, and scheduling.Table 3 evaluates these three modules through ablations averaged over three independent trials.
  • Analysis: The intrinsic latent vector zi clusters by target labels while the bias vector zb clusters by bias labels, indicating attribute disentanglement.The projections are 2D visualizations from the intrinsic and bias encoders on Colored MNIST.
  • Analysis: Randomly swapping bias features maintains reasonable intrinsic classification accuracy, indicating that Ci uses intrinsic features robustly.Table 4 reports original and swapping accuracy for intrinsic and bias classifiers across the evaluated datasets.
  • Analysis: Swapping bias features changes digit color while preserving digit shape in reconstructed Colored MNIST images.An auxiliary decoder visualizes the disentangled representation at the pixel level; reconstruction quality may be blurry because its loss is not backpropagated to classifiers.

6 Conclusions

The paper concludes that diverse bias-conflicting samples are crucial for debiasing and presents disentangled feature augmentation to create them. Experiments report state-of-the-art performance across synthetic and real-world datasets, including both Corrupted CIFAR-10 variants.

  • The method disentangles intrinsic and bias attributes, then randomly swaps their latent vectors to proliferate bias-conflicting samples.
  • Increasing bias-conflicting sample diversity is identified as crucial for debiasing.
  • The method remains superior across Corrupted CIFAR-10 Type 0 and Type 1 regardless of corruption type.

B Reconstruction of Disentangled Representation on BFFHQ

BFFHQ reconstructions test whether latent features separately encode gender and age. Swapping these features changes one attribute while preserving the other, although independently trained decoding can reduce reconstruction quality.

  • BFFHQ represents gender as the bias attribute and age as the intrinsic attribute, with age defined as either young or old.
  • Swapping bias features changes gender while preserving age, whereas swapping intrinsic features changes age while preserving gender.
  • Reconstructed examples show clear age transitions and support that z_i and z_b contain disentangled age and gender attributes.
  • The BFFHQ decoder is trained independently from the classification models, so reconstructions may be blurry or less diverse.

D.1 Datasets

The experiments use two synthetic biased datasets and one real-world face dataset with strongly correlated attributes. Evaluation uses unbiased or bias-conflicting test sets to assess debiasing.

  • Colored MNIST correlates digit labels with injected foreground colors and uses bias-conflicting ratios of 0.5%, 1%, 2%, and 5%.
  • Corrupted CIFAR-10 uses ten specified corruption types that are highly correlated with the original CIFAR-10 classes.
  • BFFHQ correlates age and gender, containing 19,200 training images—19,104 bias-aligned and 96 bias-conflicting—and 1,000 test samples.
  • Colored MNIST and Corrupted CIFAR-10 use 28×28 and 32×32 images, respectively, while BFFHQ uses 224×224 images.
  • Corrupted CIFAR-10 and BFFHQ use random crops, horizontal flips, and channel normalization; Colored MNIST uses no preprocessing augmentation.

D.3 Training Details

Training uses Adam with dataset-specific learning rates, StepLR scheduling, and scheduled feature augmentation. Reconstruction decoders are separately specified for Colored MNIST and BFFHQ.

  • Adam optimization uses learning rates of 0.01 for Colored MNIST, 0.0001 for BFFHQ, and 0.0005 or 0.001 for Corrupted CIFAR-10.
  • Colored MNIST reconstruction uses a reverse-order MLP decoder architecture.
  • BFFHQ reconstruction uses a separately documented decoder architecture and reconstruction training procedure.
  • StepLR decays every 10K iterations, with decay ratios of 0.5 for Colored MNIST and Corrupted CIFAR-10 and 0.1 for BFFHQ.
  • Feature augmentation begins after 10K iterations for all datasets.

E Broader Impacts and Limitations

Dataset bias can undermine model robustness and reliability by shifting attention from intrinsic to peripheral features, with implications for safety and social equality. The proposed approach addresses scarce bias-conflicting data but acknowledges challenges in fully disentangling real-world attributes.

  • Dataset bias makes models susceptible to peripheral rather than intrinsic features, raising distrust concerns in safety-sensitive and socially consequential applications.
  • The approach diversifies bias-conflicting features by using a large proportion of bias-aligned features, achieving state-of-the-art debiasing performance on synthetic and real-world datasets.
  • Completely disentangled representations remain difficult because bias-conflicting samples are scarce and real-world attributes can be highly correlated and complex.
Loading 2107.01372v2…