PaperlayerView exact revision on arXiv

Source-linked AI summary

A Contrastive Objective for Learning Disentangled Representations

Jonathan Kahana, Yedid Hoshen

arXiv:2203.11284v1cs.CVcs.LG

TL;DR

The paper targets image representations that are invariant to labeled domains while retaining information about other attributes. It introduces DCoDR, which combines domain-wise contrastive learning with additional constraints and initialization; across five benchmarks, it improves invariance and informativeness over the state of the art.

  • Problem

    The challenge is to learn image representations that discard labeled domain attributes while retaining information about other, unlabeled attributes.

    Source: §1 · p. 1

  • Method

    DCoDR combines a per-domain contrastive loss using only same-domain image pairs with reconstruction, image augmentations, and self-supervised pretrained encoder initialization.

    Source: §4.2 · p. 4, §8 · p. 10

  • Results

    Across five benchmarks, DCoDR significantly exceeds the state of the art in invariance and informativeness, while its discriminative variant is 5X faster than existing approaches.

    Source: §1 · p. 2

  • Takeaways & Limitations

    DCoDR provides a non-adversarial, non-generative objective for learning domain-invariant representations.

    Source: §1 · p. 2

    The per-domain invariance objective requires discrete domain labels, limiting the method's stated scope for continuous domains.

    Source: §7 · p. 10

Abstract

from arXiv · show

Learning representations of images that are invariant to sensitive or unwanted attributes is important for many tasks including bias removal and cross domain retrieval. Here, our objective is to learn representations that are invariant to the domain (sensitive attribute) for which labels are provided, while being informative over all other image attributes, which are unlabeled. We present a new approach, proposing a new domain-wise contrastive objective for ensuring invariant representations. This objective crucially restricts negative image pairs to be drawn from the same domain, which enforces domain invariance whereas the standard contrastive objective does not. This domain-wise objective is insufficient on its own as it suffers from shortcut solutions resulting in feature suppression. We overcome this issue by a combination of a reconstruction constraint, image augmentations and initialization with pre-trained weights. Our analysis shows that the choice of augmentations is important, and that a misguided choice of augmentations can harm the invariance and informativeness objectives. In an extensive evaluation, our method convincingly outperforms the state-of-the-art in terms of representation invariance, representation informativeness, and training speed. Furthermore, we find that in some cases our method can achieve excellent results even without the reconstruction constraint, leading to a much faster and resource efficient training.

1 Introduction

The paper studies representations that retain image attributes independent of a labeled domain while discarding domain information. DCoDR combines a per-domain contrastive objective with reconstruction, augmentations, and pretrained initialization to improve invariance and informativeness.

  • The goal is to retain unlabeled attributes such as pose while removing information about a labeled domain such as identity.The paper situates this task within representation disentanglement and applications including fairness and cross-domain matching.Source: §1 · p. 1
  • Domain disentanglement requires both domain invariance and informativeness about attributes independent of the domain.The paper notes that informativeness is difficult to optimize directly when attribute labels are unavailable.Source: §1 · p. 1
  • DCoDR restricts contrastive negatives to the same domain, while reconstruction, suitable augmentations, and pretrained initialization address feature suppression.The paper cautions that standard augmentations can harm invariance, and reports that pretrained initialization improves both informativeness and invariance.Source: §1 · p. 2
  • Across five benchmarks, DCoDR significantly exceeds the state of the art in invariance and informativeness.The paper also reports that its discriminative-only variant is 5X faster than existing approaches.Source: §1 · p. 2

2 Related Work

Prior work approaches disentanglement through adversarial and non-adversarial methods, while contrastive representation learning combines augmented-image pairs with a uniformity constraint. DCoDR adapts contrastive learning to the setting where domain labels are available but unlabeled factors lack sequence or evolution assumptions.

  • Disentanglement methods include adversarial training and non-adversarial approaches such as cycle consistency, group accumulation, and latent optimization.The paper positions its method as improving upon this body of work.Source: §2 · p. 2
  • Contrastive methods pair augmented samples and add a uniformity constraint to avoid representation collapse.DCoDR tailors this contrastive formulation to domain disentanglement.Source: §2 · pp. 2–3
  • Zimmerman et al. [28] address unsupervised disentanglement with assumptions about factor distributions and temporal image sequences.DCoDR instead assumes training domain labels and does not require sequences or strong assumptions about unlabeled-factor evolution.Source: §2 · p. 3
  • Disentangled representations have applications in controllable generation, image manipulation, and domain adaptation.The paper also notes anticipated impacts on interpretability, abstract reasoning, and fairness.Source: §2 · p. 3

3 Domain Invariant Representation Learning

The task assumes labeled domains and unlabeled attributes, and seeks representations that are domain-invariant yet informative about those attributes. Existing invariance approaches include adversarial domain confusion and VAE-based distribution constraints, with reported optimization and disentanglement difficulties.

3.1 Preliminaries

  • The setup provides a labeled domain and unlabeled attributes for each training image, with the domain assumed to be a categorical variable.The encoder maps each image to a representation intended to satisfy the task criteria.Source: §3.1 · p. 3

3.2 Criteria

  • The criteria require representations to be uninformative about domain while retaining as much information as possible about unlabeled attributes.Because attribute labels are unavailable, the paper describes optimizing informativeness indirectly through conditional reconstruction.Source: §3.2 · p. 3

3.3 Existing Approaches for Invariance Optimization

  • Adversarial methods use a domain discriminator to encourage invariance, but their optimization often fails to minimize the loss perfectly.VAE methods constrain representations to a normal distribution; LORD [4] found that simply optimizing this criterion did not yield disentangled representations, while latent optimization adds a time-consuming encoder-learning stage.Source: §3.3 · pp. 3–4

4 DCoDR: Learning Domain-wise Contrastive Disentangled Representation

DCoDR uses a per-domain contrastive objective to encourage domain-invariant representations, then addresses informativeness indirectly through reconstruction, image augmentations, and pre-trained encoder weights. The section also describes a faster discriminative variant and distinguishes DCoDR from SimCLR.

4.1 Overview

  • DCoDR combines per-domain contrastive invariance with indirect informativeness objectives, including reconstruction, augmentations, and pre-trained encoder initialization.Source: §4.1 · p. 4

4.2 Representation Invariance with Domain-wise Contrastive Losses

  • For each domain, DCoDR targets a uniform-sphere representation distribution; its contrastive loss uses only same-domain pairs and avoids adversarial or variational approximations.Source: §4.2 · p. 4

4.3 Improving Representation Informativeness

  • Reconstruction indirectly targets informativeness, while contrastive shortcuts can map nuisance attributes to a spherical uniform distribution and suppress other information.Source: §4.3 · pp. 4–5
  • Augmentations can improve informativeness, but poorly chosen ones can erase desired attributes; the authors select a reduced set, using Gaussian blurring alone for Edges2Shoes.Source: §4.3 · p. 5

4.4 Our Complete Method: DCoDR

  • Complete DCoDR combines the three objectives with selected augmentations and an unlabeled ImageNet MoCo-V2 encoder initialization.Source: §4.3 · p. 5, §4.4 · p. 5

4.4.1 Discriminative DCoDR (DCoDR-norec)

  • DCoDR-norec drops reconstruction and the generator, making it several times faster than DCoDR but typically less informative.Source: §4.4.1 · pp. 5–6

4.5 Differences From SimCLR

  • Unlike SimCLR, DCoDR uses separate domain-wise losses, reduced augmentations, MoCo-V2 pre-training, and a reconstruction term.Source: §4.5 · p. 6

5 Experiments

Across five benchmarks, the experiments evaluate domain invariance, informativeness, and cross-domain retrieval, including how ablations and runtime compare. DCoDR generally achieves state-of-the-art representation results, with dataset-specific limits for the discriminative variant.

5.1 Implementation Details

  • The evaluation uses five 64x64 datasets and predicts domain and other attributes from representations; CelebA landmarks are evaluated by regression L1 error.Shapes3D training uses a random 50,000-image sample; training uses a ResNet50 encoder for 200 epochs, and Edges2Shoes uses only Gaussian-blur positive augmentations.Source: §5 · p. 6, §5.2 · p. 6, §5.1 · p. 6

5.2 Representation Evaluation

  • DCoDR generally achieves state-of-the-art invariance and informativeness; both variants surpass LORD on Cars3D and perform better than LORD on CelebA, though DCoDR-norec is insufficiently informative about CelebA landmarks.On SmallNorb and Shapes3D, both variants outperform other methods on the reported disentanglement measures; CelebA has few images per domain, limiting uniform-distribution estimation.Source: §5.2 · pp. 6–7, Table 2 · p. 7

5.2.1 Ablation Study

  • Removing pre-trained initialization hurts all metrics, standard contrastive negatives hurt invariance, and unsuitable SimSiam augmentations harm both invariance and informativeness.Removing positive augmentations reduces DCoDR informativeness while improving invariance; DCoDR-norec fails to remain informative without them.Source: Table 3 · p. 7, §5.2.1 · p. 7

5.3 Cross Domain Retrieval Evaluation

  • On Cars3D, DCoDR with and without reconstruction reaches 97% retrieval accuracy; on SmallNorb, both variants retrieve better matches than baselines, while Edges2Shoes exposes a DCoDR-norec failure.On Edges2Shoes, augmentations alone do not provide enough inductive bias for informative representations, whereas reconstruction-equipped DCoDR significantly exceeds previous methods.Source: §5.3 · p. 8, Table 4 · p. 8
  • SmallNorb retrieval examples show DCoDR-norec preserving pose across varied domains, while Edges2Shoes examples show DCoDR retrieving more accurate images than DRNET and LORD.Both LORD and DCoDR-norec struggle with 180-degree flips on SmallNorb.Source: §5.3 · p. 8, Figure 2 · p. 9, Figure 3 · p. 9

5.4 Runtime Comparison

  • DCoDR-norec trains 5–10 times faster than LORD because it avoids training a generator and computing perceptual loss.The runtime comparison ran methods on one NVIDIA-RTX6000 for 200 epochs; LORD has two training stages.Source: §5.4 · p. 8, Table 5 · p. 8

6 Discussion

The discussion distinguishes conditional reconstruction from informativeness: alignment across domains is also needed. It frames pre-training and image augmentations as partial substitutes for generator-related inductive biases, while noting that all method components are needed for sufficient informativeness.

6 Discussion

  • Conditional reconstruction does not ensure informativeness unless representations are aligned across domains, with p(y|z,d) = p(y|z,d′) = p(y|z).The data-processing argument establishes the relation only for the true domain; alignment is not guaranteed without additional inductive biases, although learned representations are often well aligned in practice.Source: §6 · p. 10
  • The authors hypothesize that conditional generators help through training-induced regularization and invariance to image transformations, effects partly approximated by pre-training and augmentations.They report that all components are needed for sufficient informativeness, despite promising results from DCoDR-norec in some cases.Source: §6 · p. 10

7 Limitations

The method has three stated boundaries: it requires discrete domain labels, relies on external unsupervised pre-training, and uses image augmentations that may not transfer directly to other modalities.

7 Limitations

  • The per-domain invariance objective requires discrete domain labels, constraining the method’s domain setting.Source: §7 · p. 10
  • The method relies on external unsupervised pre-training, although the authors note the weights are publicly available and require no extra supervision for a new dataset.They compare this with prior methods that often use supervised pre-trained features in a perceptual loss.Source: §7 · p. 10
  • Image augmentations may not transfer directly to modalities such as audio or text, though the authors expect useful modality-specific augmentations can be found.Source: §7 · p. 10

8 Conclusion

DCoDR combines per-domain contrastive learning with reconstruction, image augmentations, and self-supervised pre-trained initialization to learn informative, domain-invariant image representations. The discussion of generative-model inductive biases motivates augmentation choices, including blurring, high saturation, and high contrast.

8 Conclusion

  • DCoDR combines a per-domain contrastive loss, reconstruction, image augmentations, and self-supervised pre-trained encoder initialization; its discriminative variant can be an order of magnitude faster than competing state-of-the-art methods.Source: §8 · p. 10

A.1 Investigating the Inductive Biases of Generative Models

  • The augmentation-selection approach trains unaugmented autoencoders, applies test-image transformations, and evaluates output invariance using normalized and unnormalized distance metrics.The normalized metric is sensitive to smaller transformations, while the unnormalized metric is sensitive to larger ones.Source: §A.1 · p. 14
  • Autoencoders are highly invariant to Gaussian blurring, high saturation, and high contrast, suggesting these generative-model biases may transfer to discriminative methods.Source: §A.1 · p. 14, Table 6 · p. 14

A.2 Additional Experiments

The additional experiments identify cropping as a useful augmentation for DCoDR-norec and describe the autoencoder invariance evaluation used to assess transformations. The augmentation ablation omits Shapes3D to test transfer to an unseen dataset, while acknowledging that selection could be improved.

A.2 Additional Experiments

  • On SmallNorb, cropping performed well on invariance and extremely well on informativeness, so it was added to the four-augmentation set.The set comprises Gaussian blurring, high contrast, high saturation, and cropping; the authors note dataset-specific changes may improve metrics.Source: §A.2 · p. 14, Figure 4 · p. 15

A.3 Augmentations Detailed Experimental Results

  • The detailed autoencoder evaluation reports invariance to transformations using f_unnorm and f_norm, with the individual augmentation settings specified in TorchVision-style pseudocode.The referenced table presents detailed results, but its caption alone does not establish comparative outcomes.Source: §A.3 · p. 16, Table 7 · p. 16

A.4 Other Datasets

  • Shapes3D is omitted from the augmentation ablation to test whether the selected augmentations transfer to an unseen dataset; the authors say selection could be improved.Source: §A.4 · p. 16

B Implementation Details

This section records implementation choices for reproducibility and notes that code is included. It specifies datasets, model components, and training settings.

B Implementation Details

  • The evaluation uses five datasets at 64x64 resolution, with Shapes3D subsampled as described in Section 5.The datasets are Cars3D, SmallNorb, Shapes3D, CelebA, and Edges2Shoes.Source: B Implementation Details · p. 16
  • For CelebA alone, the method excludes training classes with fewer than 20 samples, retaining 62% of classes and 81% of training samples.Other methods use the full training set; the test set is unchanged except that single-example classes are removed.Source: B Implementation Details · p. 17
  • The encoder is ResNet50, with three added fully connected layers for SmallNORB and Shapes3D; the generator uses an ImageNet-pretrained VGG perceptual loss.The generator choice is described as consistent with methods such as LORD.Source: B Implementation Details · p. 17
  • Training runs for 200 epochs with batch size 128, drawing 32 images from each of four classes.Encoder and generator learning rates are 0.0001 and 0.0003, respectively, except on CelebA, where both use 0.001.Source: B Implementation Details · p. 17
  • The contrastive temperature is 0.2 for Cars3D, SmallNORB, and CelebA, and 0.1 for Shapes3D and Edges2Shoes.Source: B Implementation Details · p. 17
  • A scalar weight of 0.3 is used for the reconstruction loss on every dataset.The weight scales reconstruction loss relative to contrastive loss.Source: B Implementation Details · p. 17

C.1 Representation Evaluation

The section points to complete representation-evaluation and retrieval results across datasets. Retrieval reports accuracies by factor and for perfect matches, with a specified tolerance for SmallNORB azimuth.

C.1 Representation Evaluation

  • Complete factor-wise representation results are provided for SmallNORB and Shapes3D; Cars3D and CelebA results were reported earlier because each predicts a single attribute.The complete results are in Tables 8 and 9 for SmallNORB and Shapes3D, respectively.Source: §C.1 · p. 17, Table 8 · p. 17, Table 9 · p. 17

C.2 Retrieval

  • Retrieval results report accuracy for each factor and perfect-match retrievals, with additional dataset results in Tables 10–12.SmallNORB, Shapes3D, and Edges2Shoes retrieval accuracies appear in Tables 10, 11, and 12, respectively.Source: §C.2 · p. 17, Table 10 · p. 18, Table 11 · p. 18
  • SmallNORB azimuth retrieval allows an error margin of 3, while all other attributes require a perfect match.Source: §C.2 · p. 17
Loading 2203.11284v1…