PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style

Julius von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf, Michel Besserve, Francesco Locatello

arXiv:2106.04619v4stat.MLcs.AIcs.CVcs.LG

TL;DR

The paper asks why hand-crafted, content-preserving augmentations yield useful self-supervised representations and formalizes this question through latent content and style variables. It proves identifiability of invariant content despite statistical and causal dependencies, tests the theory in simulations, and introduces Causal3DIdent to study practical augmentations.

  • Problem

    The paper addresses the limited theoretical understanding of why data augmentations intended to preserve semantics support effective self-supervised representation learning.

  • Method

    It models augmentation as paired views from a latent-variable model partitioned into invariant content and changing style, allowing statistical and causal dependencies.

  • Results

    The paper proves sufficient generative and discriminative conditions for identifying the invariant content partition, finds simulations consistent with the theory, and introduces Causal3DIdent for systematic augmentation analysis.

  • Takeaways & Limitations

    Under the paper’s assumptions, self-supervised learning with augmentation can isolate what remains invariant across views rather than requiring independent latent factors.

  • Takeaways & Limitations

    The choice of augmentation implicitly defines content and style, and image-level transformations may affect underlying ground-truth factors in unanticipated ways.

Abstract

from arXiv · show

Self-supervised representation learning has shown remarkable success in a number of domains. A common practice is to perform data augmentation via hand-crafted transformations intended to leave the semantics of the data invariant. We seek to understand the empirical success of this approach from a theoretical perspective. We formulate the augmentation process as a latent variable model by postulating a partition of the latent representation into a content component, which is assumed invariant to augmentation, and a style component, which is allowed to change. Unlike prior work on disentanglement and independent component analysis, we allow for both nontrivial statistical and causal dependencies in the latent space. We study the identifiability of the latent representation based on pairs of views of the observations and prove sufficient conditions that allow us to identify the invariant content partition up to an invertible mapping in both generative and discriminative settings. We find numerical simulations with dependent latent variables are consistent with our theory. Lastly, we introduce Causal3DIdent, a dataset of high-dimensional, visually complex images with rich causal dependencies, which we use to study the effect of data augmentations performed in practice.

1 Introduction

The paper studies why self-supervised learning with semantic-preserving augmentations works by modeling invariant content and changing style, without assuming latent independence. It proves content-partition identifiability, validates the theory under dependencies, and introduces Causal3DIdent for analyzing practical augmentations.

  • Motivation: Data augmentation is empirically effective but theoretically incompletely understood, motivating an identifiability analysis of representations learned from augmented views.The paper frames augmentation as weak supervision intended to preserve semantics while changing nuisance aspects.
  • Problem formulation: The model allows statistical and causal dependencies between latent variables instead of assuming mutually independent factors.This extends beyond assumptions common in independent component analysis and disentanglement.
  • Problem formulation: The latent representation is partitioned into invariant content and augmentation-varying style, matching the intended semantic role of data augmentation.Only style is assumed to change between original and augmented views.
  • Contributions: The paper proves that generative and discriminative self-supervised learning can identify the invariant content partition up to the stated representation ambiguity.The discriminative results cover invertible encoders and non-invertible encoders with entropy regularisation.
  • Empirical validation: Simulations with dependent content and style variables agree with the theory, while Causal3DIdent supports systematic study of what practical augmentations retain or discard.The dataset contains high-dimensional 3D-object images with causal dependencies.

2 Preliminaries and background

The background describes augmentation-based self-supervised learning as training encoders on pairs of transformed views, typically enforcing representation similarity while preventing collapse. It situates the paper’s identifiability question against prior multi-view methods that usually allow all independent factors to change.

  • Self-supervised learning with data augmentation: Self-supervised augmentation samples transformations such as crops, flips, blur, colour distortion, or cutouts and applies separate draws to each observation.The resulting pair of views is used as the learning signal.
  • Joint embedding: Joint-embedding methods map augmented views through one or two encoders and train their representations to be close under a similarity metric.Cosine similarity and negative L2 norm are cited examples.
  • Contrastive learning: Contrastive learning pulls positive augmented pairs together and pushes representations of negative pairs apart, preventing a constant collapsed representation.InfoNCE combines these alignment and separation objectives.
  • Contrastive learning: InfoNCE can be interpreted through alignment and uniformity, with its denominator acting as a nonparametric entropy estimator as the number of negatives grows.The objective also lower-bounds mutual information across similar views.
  • Identifiability: Prior identifiability approaches often assume independent factors and use changes across views, environments, or time, whereas this paper asks whether augmentation can reveal invariant content.The general nonlinear i.i.d. case is not identifiable without additional assumptions.

3 Problem formulation

The paper models observations through an invertible latent-variable generator and represents augmentation as a stochastic latent-space change. Content remains fixed, while a possibly partial subset of style variables changes, including under causal dependence of style on content.

  • Augmentation model: Augmentation is modeled by modifying z through a conditional distribution and applying the same rendering function to obtain the augmented observation.This representation-centric view corresponds to observation-level transformations through the inverse generator.
  • Generative model: Observations are generated as x = f(z), where a continuous latent variable is mapped to the observation space by a smooth invertible function.The latent density is defined on an open, simply-connected representation space.
  • Content and style: The latent code is partitioned into content c and style s, with content shared across views and style permitted to vary.The dimensions satisfy n = n_c + n_s and the latent space is expressed as C × S.
  • Style changes: At each augmentation, only a random subset of style variables needs to change, allowing transformations to affect some style properties while leaving others unchanged.This is weaker than requiring every style variable to change.
  • Causal interpretation: A causal formulation permits content to cause style, and augmentation acts as a soft intervention that perturbs style while holding factual exogenous variables fixed.The resulting augmented observation is interpreted as a counterfactual under the modified style mechanism.

4 Theory: block-identifiability of the invariant content partition

The paper defines block-identifiability as recovering the invariant content partition up to an invertible mapping, without requiring independent latent factors. It proves this recovery for generative models and discriminative encoders, including non-invertible encoders with entropy regularisation.

  • Definition and scope: Block-identifiability recovers all and only content information up to an invertible mapping, rather than identifying individual latent factors.This notion targets content–style separation under dependencies within the latent representation.
  • Definition and scope: The theory allows nontrivial statistical and causal dependencies between latent variables, unlike independence-based approaches such as ICA and disentanglement.The causal setting permits content to influence style while remaining invariant across augmentations.
  • Generative learning: Generative maximum-likelihood models block-identify content when the assumed latent process, smooth invertible decoder, positive latent density, and style-change conditions hold.The result is asymptotic: with infinite data, the recovered content partition matches the true one up to an invertible function.
  • Generative learning: Generative identifiability follows because view invariance first constrains the learned representation and then excludes dependence on changing style variables.The contradiction argument uses the augmentation conditions to rule out style information in the invariant block.
  • Discriminative learning: For invertible discriminative encoders, aligning the first n_c representation dimensions is sufficient to isolate the invariant content partition.Invertibility preserves all information and therefore prevents collapse under the alignment objective.
  • Discriminative learning: For non-invertible encoders, adding maximum-entropy regularisation to alignment removes the invertibility requirement and still block-identifies content.The entropy term prevents collapse by favoring preservation of content information; this result is closest to common contrastive SSL practice.

5 Experiments

The experiments test the theory under dependent latent variables and use Causal3DIdent to examine what practical augmentations preserve as content or discard as style. Simulations support block-identification, while image-level augmentations affect latent factors in augmentation-specific ways.

  • Experimental design: The experiments combine a controlled numerical test of Thm. 4.4 with a systematic study of practical data augmentations on Causal3DIdent.The numerical setting includes statistical dependence within content and style blocks and causal dependence of style on content.
  • 5.1 Numerical data: Across four increasingly complex generative processes, content is block-identified, including settings with statistical and causal dependencies.An R2 close to one indicates that almost all variation is explained by the learned content representation.
  • 5.2 High-dimensional images: Causal3DIdent: Image-level augmentations can fail to leave specific latent factors invariant, and their effects generally resemble varying corresponding groups of latent variables.Colour distortion discards hue, crops discard position, and rotation affects both object rotation and position.
  • 5.2 High-dimensional images: Causal3DIdent: Class is reliably decoded only when crops and colour distortion are combined, while augmentations yielding good classification vary the other latents.The combined augmentation mirrors the reported ImageNet evaluation pattern.
  • Additional analyses: The synthetic ablation finds insufficient content capacity produces a lower-dimensional content mixture, whereas excess capacity encodes some style information.These effects are described for dim(ĉ) below or above the content dimension nc.

6 Discussion

The discussion emphasizes that practical augmentations may violate theoretical assumptions and that augmentation choices determine which factors become content or style. It also identifies open questions about extending the theory beyond entropy-based regularization.

  • Theory vs practice: Some technical assumptions required for block-identifiability may not hold exactly in practice, as illustrated by the Causal3DIdent experiments.Class can be invariant in principle yet fail to be reliably encoded under some image-level augmentations.
  • Theory vs practice: Image-level augmentations can introduce variation outside the generative image manifold and change multiple latent factors in correlated ways.Colour distortion may create three-dimensional colour variation although Causal3DIdent contains only one hue degree of freedom.
  • Theory vs practice: The theory assumes continuous latents and holds asymptotically at the global optimum, whereas practice uses finite samples and non-convex optimization.The authors identify relaxing the continuity assumption as a possible direction for future work.
  • Theory vs practice: Using two augmented views can help because different changed style subsets increase variability across the view pair.If the views change subsets A and A′, their difference involves the union A ∪ A′ almost surely.
  • Beyond entropy regularisation: The connection between entropy maximization and architectural regularizers in BYOL or SimSiam remains open.The discussion explicitly contrasts these approaches with the established connection to SimCLR and the intuitive connection to BarlowTwins.
  • Practical implications: Because augmentation choices define the content-style partition, practitioners should use caution when applying them to safety-critical domains or identifying specific latent properties.Style may still correlate with a downstream task, even though content-only representations can be optimal for arbitrary tasks where style changes adversarially.
  • Conclusion: The work claims the first identifiability result under arbitrary latent dependence and an empirical augmentation study covering classification and continuous ground-truth latents.The approach uses invariance—what is always shared across views—as a learning signal.

Funding Transparency Statement

The paper acknowledges financial and institutional support and describes the contents of its appendices.

  • The work received support from DFG, Open Philanthropy, Good Ventures Foundation, BMBF, and the Machine Learning Cluster of Excellence.
  • The appendices contain full proofs, additional Causal3DIdent details and plots, experimental analyses, and implementation details.
  • The appendix proofs build on one another, with a main step from Thm. 4.2 reused in the proofs of Thms. 4.3 and 4.4.

A.1 Proof of Thm. 4.2

The proof shows that any matching-likelihood generative model satisfying the stated smoothness, density, and augmentation assumptions identifies invariant content independently of style, up to a smooth invertible mapping.

  • Assumptions: Theorem 4.2 assumes a smooth invertible mixing function, positive smooth latent density, and changing-style subsets with locally supported conditional distributions.These conditions are stated as assumptions (i)–(iii) for the generative model.
  • Step 1: Matching likelihoods first imply that the learned representation is related to the true latent code by a smooth invertible mapping and is invariant in its content coordinates.The invariance holds almost surely under the true generative process.
  • Step 2: The proof then assumes style dependence in the learned content coordinates and constructs an open region where the invariance discrepancy is strictly positive.Strict monotonicity in a style variable produces the discrepancy, while continuity makes the positive-discrepancy region open.
  • Conclusion: Therefore, the learned content coordinates depend only on true content and are related to it by a smooth invertible mapping.This establishes block-identification of the true content variables.

A.2 Proof of Thm. 4.3

The proof establishes that minimizing the alignment loss with a smooth invertible encoder forces invariance in the learned content coordinates and therefore block-identifies true content.

  • Setup: Theorem 4.3 considers any smooth invertible encoder minimizing the alignment functional under the same generative assumptions as Theorem 4.2.The result concerns global minimizers of the alignment objective.
  • Step 1: The alignment loss is nonnegative and equals zero exactly when the first content coordinates agree across every observed view pair with positive density.The true inverse mixing function attains this global minimum.
  • Step 1: Any zero-loss encoder induces a smooth invertible transformation of the true latent variables whose first coordinates are invariant across augmented views.The proof rewrites the encoder as g = h ◦ f^-1 and obtains the same invariance condition used in Theorem 4.2.
  • Step 2: Reusing Theorem 4.2’s contradiction argument shows that these invariant coordinates can depend only on true content, not style.Thus the encoder’s learned content block is related to the true content block through an invertible mapping.

A.3 Proof of Thm. 4.4

The proof shows that discriminative learning with a non-invertible encoder can identify content by combining view alignment with maximum entropy under the paper’s assumptions.

  • Setup: Theorem 4.4 considers smooth encoders mapping observations to (0, 1)^nc that minimize an alignment-plus-entropy functional.This setting does not require the encoder itself to be invertible.
  • Step 1: The global minimum requires zero alignment loss and maximum entropy, whose unique distribution on (0, 1)^nc is uniform without additional moment constraints.The proof first constructs a smooth content transformation with a uniform output distribution.
  • Step 1: The constructed transformation attains the global minimum because content agrees across views and the uniform distribution on the unit hypercube has zero entropy.The claim is verified using the true inverse mixing function restricted to content coordinates.
  • Steps 1–2: Any minimizer is invariant across views and uniformly distributed, while the repeated contradiction argument removes dependence on style variables.The resulting representation depends only on true content.
  • Step 3: A proposition on smooth maps between simply connected manifolds then establishes invertibility of the content mapping.Regularity of the true content density and its uniform pushforward supplies the proposition’s conditions.

B.3 Dataset Visuals

Causal3DIdent visualizations and tables describe class-conditional image samples and dependencies among object, lighting, position, rotation, and hue variables.

  • Class marginals: The dataset visual section shows 40 random samples from each object class’s marginal distribution.The displayed classes include Teapot, Hare, Dragon, Cow, Armadillo, Horse, and Head.
  • Rotation: Table 2 records that the center of the truncated normal rotation distribution varies with object class.This makes rotation class-dependent in the dataset.
  • Position: Table 3 records that the center of the truncated normal xy-position distribution varies with object class and spotlight position.Spotlight position is rescaled from [−1, 1] to [−π/2, π/2].
  • Hue: Table 4 records that object-hue distribution centers vary with object class, background hue, and spotlight hue.For Hare and Dragon classes, the object can blend into or stand out from the environment.
  • Experiments: The appendix describes additional nonlinear and linear evaluations, ablations, and experiments on Causal3DIdent and MPI3D-real.These include SimCLR and BarlowTwins evaluations.

C.1 Numerical Data

The numerical experiments test whether the learned representation identifies content under statistical and causal dependence. Results support block-identification while showing that dependence can make style predictable from content and affect dimensionality requirements.

  • C.1 Numerical Data: Linear regression still reveals block-identification of content across four increasingly complex generative processes.The evaluation reports mean ± standard deviation R2 over three random seeds.
  • C.1 Numerical Data: Varying the learned representation dimension changes identifiability: insufficient capacity mixes content, while excess capacity encodes some style.When dim(ĉ) < nc, not all content fits; when dim(ĉ) > nc, entropy favors encoding style information.
  • C.1 Numerical Data: Statistical dependence substantially improves style-identification scores and can increase content-identification scores at small representation dimensions.At dim(ĉ) = 1, content R2 is approximately 0.4 with statistical dependence versus 0.2 without it.
  • C.1 Numerical Data: Because style is linearly dependent on content, a representation that block-identifies content can still predict aspects of style.This appears as a relatively large R2 for style in the causally dependent setting.
  • C.1 Numerical Data: The experiments conclude that latent dependence may reduce effective dimensionality, causing style to be encoded alongside content unless representation size is reduced.Content isolation in the block-identifiability sense does not prevent style predictability when content and style are causally dependent.

C.2 Causal3DIdent

Causal3DIdent experiments examine how practical image augmentations shape encoded content across object factors. They find augmentation effects depend on the transformation combination, representation layer, and dataset, with several empirical limitations relative to the theory.

  • C.2 Causal3DIdent: Rotation combined with crops produces results similar to crops alone, suggesting position and rotation are difficult to disentangle in Causal3DIdent.This relationship holds for image-level augmentations and corresponding latent transformations.
  • C.2 Causal3DIdent: Combining rotation with small crops and colour distortion preserves background hue while slightly decreasing object position and rotation scores.Adding a third augmentation lessens the individual effects because colour distortion discourages background-hue encoding while the other augmentations encourage it.
  • C.2 Causal3DIdent: Linear evaluation reduces R2 scores overall, but the augmentation patterns observed in the full evaluation persist.The results are reported for class and all continuous variables.
  • C.2 Causal3DIdent: Intermediate-layer representations retain factors discarded by the final layer; for example, z-position reaches R2 = 0.88 there versus R2 = 0.15 in the final layer under hue-changing latent transformations.All R2 scores increase across the board for the intermediate representation.
  • C.2 Causal3DIdent: Intermediate-layer recovery does not remove the discrepancy between practical image augmentations and latent transformations.The paper reports that intermediate features contain more transformation information, while no one-to-one correspondence can be made between the two augmentation types.
  • C.2 Causal3DIdent: BarlowTwins reproduces the main augmentation trends across two redundancy-reduction weights, including hue discarding under colour distortion and crop-dependent object-hue identifiability.The reported settings are λ = 0.0051 and λ = 0.051.
  • C.2 Causal3DIdent: MPI3D-real shows a similar trend: colour distortion decodes colour factors worse than positional and rotational information.The comparison is subject to the dataset’s stated caveats and uses dim(ĉ) = 5.

D Experimental details

The experiments instantiate dependent latent-variable models, train encoders with contrastive objectives, and evaluate whether learned representations recover content and style. Practical details cover synthetic simulations, Causal3DIdent, augmentations, regression evaluation, and compute.

  • D Experimental details: Synthetic data generation permits statistical dependence within content and style blocks and causal dependence of style on content.Dependence is controlled through covariance matrices and the conditional model s|c ∼ N(a + Bc, Σs).
  • D Experimental details: The encoder is trained on paired views with InfoNCE using negative L2 similarity and empirical averages with negative samples.The setup approximates the entropy-regularized theoretical objective.
  • D Experimental details: Causal3DIdent uses a ResNet18 encoder with a 100-dimensional hidden layer, trains for 200,000 iterations, and evaluates on 25,000 held-out samples.Training uses Adam with learning rate 10^-4.
  • D Experimental details: The image-level augmentation study includes crops, colour distortion, random rotation, and their visual effects on representative samples.Figure 11 shows original images, small and large crops with flips, colour distortion, and random rotation.
  • D Experimental details: Content identifiability is evaluated by predicting ground-truth content from ĉ = g(x) with kernel ridge regression.The Gaussian kernel is universal, supporting nonparametric approximation of the invertible mapping in the theorem.
  • D Experimental details: The evaluation does not use MCC or MIG because the theory allows dependence and targets block-identification rather than one-to-one factor correspondence.The paper instead treats identification of the invertible mapping h as a regression task.
  • D Experimental details: Synthetic experiments require roughly 5–10 hours on one RTX 2080 Ti, while Causal3DIdent experiments take 28 hours on four such GPUs.Dataset creation requires approximately 150 additional GPU-hours on one RTX 2080 Ti.
Loading 2106.04619v4…