PaperlayerView exact revision on arXiv

Source-linked AI summary

Disentanglement of Correlated Factors via Hausdorff Factorized Support

Karsten Roth, Mark Ibrahim, Zeynep Akata, Pascal Vincent, Diane Bouchacourt

arXiv:2210.07347v3cs.LGstat.ML

TL;DR

Unsupervised disentanglement methods commonly assume statistically independent factors, despite correlations in realistic data. This paper proposes HFS, which encourages pairwise factorized support using a Hausdorff distance. Across benchmarks and correlation settings, HFS improves disentanglement, with reported gains in classification robustness to factor shifts as well.

  • Problem

    Common unsupervised disentanglement methods assume statistically independent factors, although realistic data contains correlations and prior work reports failures under such correlations.

    Source: §1 · p. 2, §2.1 · p. 3

  • Method

    HFS relaxes independence to pairwise factorized support and encourages it by minimizing a Hausdorff distance between latent support and its factorization.

    Source: §1 · p. 2, §2.3 · p. 4

  • Results

    HFS improves DCI-D disentanglement by up to +61% over standard methods and classification robustness to factor shifts by up to +28%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The criterion is intended to accommodate distribution shifts within factorized support without restricting the distribution over that support to independence.

    Source: §1 · p. 2

    The assumption still requires every combination of individually supported factor values to be possible, even when factor distributions are dependent.

    Source: §2.2 · p. 3

Abstract

from arXiv · show

A grand goal in deep learning research is to learn representations capable of generalizing across distribution shifts. Disentanglement is one promising direction aimed at aligning a model's representation with the underlying factors generating the data (e.g. color or background). Existing disentanglement methods, however, rely on an often unrealistic assumption: that factors are statistically independent. In reality, factors (like object color and shape) are correlated. To address this limitation, we consider the use of a relaxed disentanglement criterion -- the Hausdorff Factorized Support (HFS) criterion -- that encourages only pairwise factorized \emph{support}, rather than a factorial distribution, by minimizing a Hausdorff distance. This allows for arbitrary distributions of the factors over their support, including correlations between them. We show that the use of HFS consistently facilitates disentanglement and recovery of ground-truth factors across a variety of correlation settings and benchmarks, even under severe training correlations and correlation shifts, with in parts over $+60\%$ in relative improvement over existing disentanglement methods. In addition, we find that leveraging HFS for representation learning can even facilitate transfer to downstream tasks such as classification under distribution shifts. We hope our original approach and positive empirical results inspire further progress on the open problem of robust generalization. Code available at https://github.com/facebookresearch/disentangling-correlated-factors.

1 INTRODUCTION

The introduction argues that disentanglement may aid generalization, but common unsupervised methods assume ground-truth factors are independent despite real-world correlations. It proposes pairwise Hausdorff Factorized Support (HFS) as a weaker criterion and reports improvements in disentanglement and downstream classification under shifts.

1 INTRODUCTION

  • Disentanglement is framed as useful for reliable in- and out-of-distribution generalization, and defined as a one-to-one correspondence between each ground-truth factor and a representation dimension.The introduction also identifies interpretability and fairness as potential benefits (Locatello et al., 2019a; Träuble et al., 2021).Source: §1 · p. 1
  • Existing unsupervised disentanglement methods assume independent ground-truth factors, although real data contains correlations associated with failures in prior evaluations.The paper also notes that fully unsupervised disentanglement is theoretically impossible in the general unconstrained nonlinear case; practical results rely on architectural inductive biases (Rolinek et al., 2019).Source: §1 · p. 2
  • HFS replaces factor independence with factorized support, allowing arbitrary distributions over that support, including correlated factors.The introduction further narrows the criterion to pairwise factorized support for tractability and notes an independent causal derivation in Wang & Jordan (2021).Source: §1 · p. 2
  • The proposed pairwise HFS criterion encourages factorized latent support by minimizing a Hausdorff distance between the finite-sample support and its factorization.The criterion is also presented as a regularizer that can be combined with existing disentanglement methods.Source: §1 · p. 2
  • HFS improves DCI-D disentanglement by up to +61% over standard methods and classification robustness to factor shifts by up to +28%, with improved sample efficiency.The reported experiments span three benchmarks and up to 14 correlation settings.Source: §1 · p. 2

2 PROPOSED APPROACH

The approach replaces the usual independence assumption with factorized support, allowing correlated factors while retaining a criterion for disentanglement. It operationalizes that criterion with Hausdorff distance and combines it with autoencoder objectives.

2.1 DISENTANGLEMENT VERSUS INDEPENDENCE

  • Unsupervised disentanglement methods often equate disentanglement with statistically independent representation components, although realistic factors may be dependent.The paper distinguishes recovering ground-truth factors from imposing a factorial prior on their representation.Source: §2.1 · p. 3

2.2 RELAXING THE INDEPENDENCE ASSUMPTION INTO THAT OF FACTORIZED SUPPORT

  • Factorized support requires every combination of individual factor values to be possible, but does not require those combinations to be equally likely.Independence implies factorized support, but factorized support does not imply independence.Source: §2.2 · p. 3

2.3 A PRACTICAL CRITERION FOR FACTORIZED SUPPORT

  • The method measures the gap between the representation's support and the Cartesian product of its marginal supports with a Hausdorff distance.It further encourages pairwise factorized support, using a pairwise Hausdorff estimate to keep computation tractable in high dimensions.Source: §2.3 · pp. 3–4
  • A finite-batch approximation estimates the supports from batch representations, while an autoencoder reconstruction term prevents representations from collapsing to a single point.Larger tuples are possible, but their computational and statistical benefits shrink.Source: §2.3 · p. 4, §2.4 · p. 4

2.4 PUTTING IT ALL TOGETHER

  • The HFS objective combines the stochastic autoencoder loss with the Hausdorff estimate, and the Hausdorff term can also regularize existing VAE-based methods.In the plain HFS objective, the Hausdorff term replaces the β-VAE KL term and removes the factorial prior.Source: §2.4 · pp. 4–5

3 RELATED WORK

Related work connects disentanglement to downstream transfer, while noting limitations of independence-based methods under factor correlations. This paper positions pairwise factorized support as a generic alternative and uses DCI-D as its leading evaluation metric.

Disentanglement under correlated factors

  • Studies of correlated-factor settings report that unsupervised methods assuming independent factors can fail to disentangle, with potentially negative impact on OOD generalization (Montero et al., 2021; 2022; Träuble et al., 2021; Dittadi et al., 2021).Prior setups include artificial correlations and excluding factor-value combinations.Source: §3 · p. 5
  • The paper proposes a generic remedy without auxiliary variables or prior models: pairwise factorized support, derived by relaxing factor independence.It contrasts this with a related support-factorization idea derived from a causal perspective by Wang & Jordan (2021), and reports better performance and scaling for its pairwise relaxation.Source: §3 · pp. 5–6
  • The authors use DCI-D as their leading disentanglement metric, citing evidence that it may be more suitable than alternatives (Locatello et al., 2020a; Dittadi et al., 2021).They also note generally strong correlation between metrics (Locatello et al., 2019b).Source: §3 · p. 5

4 EXPERIMENTS

Across three benchmarks, HFS promotes disentanglement under correlated factors; on Shapes3D, it also improves transfer across correlation shifts and downstream classification, while support factorization tracks test disentanglement.

4 EXPERIMENTS

  • Experiments extend correlated-factor benchmarks to correlation shifts and downstream transfer, using at least six seeds per reported result and at least seven hyperparameter settings per baseline.Key experiments use at least ten seeds; the correlation framework includes multiple factor pairs and shared confounders.Source: §4 · p. 6

4.1 FACTORIZATION OF SUPPORTS FOR DISENTANGLEMENT ON STANDARD BENCHMARK

  • HFS matches or outperforms comparable β-VAE settings and improves DCI-D by nearly +30% without correlations, over +60% in some shared-confounder settings, and up to +140% over β-TCVAE.Results cover Shapes3D, MPI3D, and DSprites, with test performance measured on uncorrelated data.Source: §4.1 · p. 6, Table 1 · p. 7

4.2 OUT-OF-DISTRIBUTION GENERALIZATION UNDER CORRELATION SHIFTS

  • On Shapes3D, β-VAE + HFS improves test disentanglement across all correlation shifts, while downstream classification gains from HFS grow as shifts become larger.The β-VAE baseline’s disentanglement drops with increased training correlation; its classification performance also drops with higher training correlation or greater test variation.Source: §4.2 · pp. 7–8, Figure 2 · p. 8

4.3 BENEFITS UNDER VARYING DOWNSTREAM ADAPTATION METHODS

  • HFS improves downstream classification in all tested settings, with stronger low-data gains for full latents and nearly +80% relative improvement using the most expressive single entry.In the standard uncorrelated setting, relative improvements increase from 4% to 45% as labeled training data decreases.Source: Figure 3 · p. 8, §4.3 · p. 9

4.4 FACTORIZATION AS A PERFORMANCE METRIC

  • As HFS weighting increases, lower training-set support-factorization estimates accompany higher test disentanglement; this relationship persists under stronger training correlations.Unlike HFS, β-VAE’s implicit push toward independence hurts disentanglement and generalization in these experiments.Source: §4.4 · p. 9, §4.3 · p. 9, Figure 4 · p. 9

5 CONCLUSION

HFS relaxes independence to pairwise factorized support, allowing correlated distributions over the support. The paper reports improved disentanglement across correlation shifts and improved out-of-distribution generalization.

5 CONCLUSION

  • HFS improves disentanglement across correlation shifts, with relative gains over 60% on common benchmarks, and the gains are reflected in improved out-of-distribution generalization.The reported generalization improvement is especially apparent as distribution shifts become more severe.Source: §5 · p. 9
  • The method uses pairwise Hausdorff set-distances to encourage factorized support and can steer existing disentanglement methods toward that support.The paper says this criterion allows arbitrary distributions over the support, including correlated factors.Source: §5 · p. 9
  • The approach avoids requiring statistically independent factors, instead aiming to recover a factorized support.Source: §5 · p. 9

REPRODUCIBILITY STATEMENT

The appendix provides reproduction guidance and evaluates alternatives to the pairwise Hausdorff approximation, including averaged distances, subsampling, and soft selection.

REPRODUCIBILITY STATEMENT

  • The authors released their codebase and direct readers to detailed correlation settings, training, model, and grid-search information for reproducing the results.Transfer-experiment training and correlation settings are the same as those used earlier in the paper.Source: REPRODUCIBILITY STATEMENT · p. 10

A ALTERNATE HAUSDORFF VARIANTS

A.1 AVERAGED HAUSDORFF

  • The averaged Hausdorff alternative replaces the max operation with an average to reduce sensitivity to outliers.Source: §A.1 · p. 16

A.2 SUBSAMPLING

  • Pairwise approximations worked better for optimization than subsampling the full high-dimensional factorized support, which optimizes a single selected distance pair.The pairwise approach provides training gradients across multiple latent subsets.Source: §A.2 · p. 16

A.3 SAMPLING-BASED SOFTMIN

  • Sampling-based softmin exploration offered minimal benefits in the experiments.The proposed selection samples a final distance to minimize, allowing a controllable degree of exploration during training.Source: §A.3 · p. 16

A.4 SOFTENED HAUSDORFF DISTANCE

The softened Hausdorff variants relax the hard minimum and maximum operations. The reported results favor soft settings that converge toward the default hard objective.

A.4 SOFTENED HAUSDORFF DISTANCE

  • The comparison considered replacing max with averaging, probabilistic approximation of min, and subsampling the full-dimensional support, using at least five seeds and a grid search over γ.The tested γ values were 1, 3, 10, 30, 100, 300, 1000, 3000, and 10000.Source: §A.4 · p. 17
  • Soft approximations generally performed best when converging toward the default hard variant; large outer temperatures approached the mean approximation.Source: Figure 5 · p. 17
  • The inner temperature controls emphasis on the minimal distance versus more uniform weighting, while the outer temperature controls emphasis on the maximum versus more uniform weighting over soft distances.As the outer temperature approaches infinity, the weighting approaches a mean over soft distances.Source: §A.4 · p. 17

B EVALUATION OF HAUSDORFF DISTANCE APPROXIMATION VARIANTS

This appendix compares approximations to the Hausdorff objective and examines scale regularization. Pairwise approximations perform well, while extra scale regularization does not show consistent gains across correlation settings.

B EVALUATION OF HAUSDORFF DISTANCE APPROXIMATION VARIANTS

  • Pairwise sliced approximations outperform subsampling the full high-dimensional support for optimization, providing gradients across latent subsets rather than optimizing one selected distance pair.Source: B EVALUATION OF HAUSDORFF DISTANCE… · pp. 17–18
  • Disentanglement performance rose from 64% to 75% in the uncorrelated transfer setting as multiple 2D approximations were used, with diminishing returns as more pairs were included.Source: B EVALUATION OF HAUSDORFF DISTANCE… · p. 18, Table 3 · p. 18
  • Probabilistic min-selection and replacing the outer max offered no notable benefits, and replacing the outer max can severely impact disentanglement performance.Source: B EVALUATION OF HAUSDORFF DISTANCE… · p. 18
  • Soft variants performed best with small temperatures approximating the hard variant, but showed no major benefits and introduced two additional hyperparameters.Performance consistently dropped when either temperature was reduced, and increasing the outer temperature moved performance toward the mean variant.Source: B EVALUATION OF HAUSDORFF DISTANCE… · p. 18
  • The authors choose 25 latent pairs as a balance between performance and compute cost; adding HFS raises VAE epoch time to 60s versus 52s for β-VAE, below β-TCVAE at 70s and FactorVAE at 96s.The training-time impact diminishes with larger backbone networks.Source: B EVALUATION OF HAUSDORFF DISTANCE… · p. 18

C REGULARIZING SCALE TO AVOID COLLAPSE

D HYPERPARAMETER EVALUATION

HFS weighting γ is less sensitive to its exact setting than the β-VAE prior-matching weight β. The passage attributes this robustness to factorizing support rather than imposing distribution-level matching.

D HYPERPARAMETER EVALUATION

  • Figure 6 characterizes the robustness of training dynamics to factorized-support weighting γ as much less detrimental overall.Source: Figure 6 · p. 19
  • HFS support factorization is much less dependent on the weighting γ than β-VAE is on the KL-divergence weight β.The comparison uses grid searches over both weights.Source: D HYPERPARAMETER EVALUATION · p. 19
  • The authors attribute this lower sensitivity to support factorization being a weaker constraint on training dynamics than matching distributions.They also describe support factorization as a more realistic property.Source: D HYPERPARAMETER EVALUATION · p. 19

E SAMPLE RECONSTRUCTIONS AND QUALITATIVE EVALUATION

Qualitative traversals suggest that HFS can help separate factors that β-VAE struggles to disentangle, including under training correlations and in uncorrelated failure cases. The appendix extends the evaluations with detailed correlation settings, adaptation results, and additional support-factorization analyses.

E SAMPLE RECONSTRUCTIONS AND QUALITATIVE EVALUATION

  • Across correlated training data and uncorrelated β-VAE failure cases, explicit factorization of support helps separate factors β-VAE initially struggles to disentangle, beyond increasing maximally achievable DCI-D.Figure 7 shows Shapes3D sample traversals; each setup uses its best seed out of 10, and each image reports overall DCI-D for β-VAE and β-VAE + HFS.Source: E SAMPLE RECONSTRUCTIONS AND… · p. 19, Figure 7 · p. 20

F DETAILED FIGURES AND TABLES

F.1 ADDITIONAL DISENTANGLEMENT RESULTS

  • The extended Table 5 studies HFS as both a standalone objective and a regularizer across training-correlation settings.It includes detailed splits and additional settings; the associated pairings are indexed in §H.Source: §F.1 · p. 21, Table 5 · p. 21

F.2 FURTHER ADAPTATIONS

  • With an L1-regularized linear probe, relative ground-truth-factor classification improvements from β-VAE + HFS increase with training-correlation severity.Figure 8 reports average performance across different amounts of test data, using an L1-optimal probe.Source: §F.2 · pp. 21–22, Figure 8 · p. 21

F.3 FURTHER PROGRESSIONS

  • Figure 9 relates training-support factorization to improved test disentanglement and shows that β-VAE’s factorial-prior KL objective implicitly minimizes for factorized support across settings.Source: §F.3 · p. 22, Figure 9 · p. 22

F.4 DETAILED CORRELATION SHIFT TRANSFER GRID

  • The appendix supplies the exact correlation settings used in the transfer grid for Figure 2.These settings are provided for replicability.Source: §F.4 · p. 22, Figure 10 · p. 22

F.5 DETAILED METRIC GRID

The appendix details a metric-transfer grid and describes the disentanglement measures used in the paper. DCI-D is the leading disentanglement measure, while DCI-I measures generalization performance.

F.5 DETAILED METRIC GRID

  • Figure 11 relates training-data Hausdorff distance to factorized support with DCI and MIG across correlation shifts, and reports a strong relation to downstream disentanglement.Figure 4’s right-hand metric-transfer grid is also reproduced with exact correlation settings.Source: §F.5 · p. 22, Figure 11 · p. 23

G.1 DCI-D AND DCI-I

  • DCI-D measures whether representation dimensions predict single factors, while DCI-I is average test-split prediction error.The metrics use models trained to predict ground-truth factors from representations; DCI-D weights dimension-wise entropy scores by overall predictive importance.Source: §G.1 · p. 23

G.2 MUTUAL INFORMATION GAP (MIG)

  • MIG averages, across factors, the normalized mutual-information gap between each factor’s two most informative representation entries.The implementation estimates discrete mutual information by binning each representation dimension into 20 bins over 10,000 samples.Source: §G.2 · pp. 23–24

G.3 MODULARITY

  • Modularity evaluates whether each representation dimension depends on at most one ground-truth factor, using normalized squared mutual-information scores.The implementation discretizes mutual information by binning each latent entry into 20 bins over 10,000 samples.Source: §G.3 · p. 24

G.4 SAP SCORE

  • SAP averages each factor’s prediction-error gap between its two most predictive latent entries, using linear prediction models.The cited implementation uses a linear SVM with C = 0.01, 10,000 training samples, and 5,000 test points.Source: §G.4 · p. 24

G.5 BETA- AND FACTORVAE SCORES

  • The β-VAE score classifies the fixed factor from average absolute differences between observation sets; FactorVAE instead votes using the least-variable representation entry.FactorVAE is described as an extension of the β-VAE score; Table 6 lists hyperparameter grid searches for baselines and the factorized-support objective.Source: §G.5 · p. 24, Table 6 · p. 25

H FURTHER EXPERIMENTAL DETAILS

The appendix records the experimental implementation, correlation configurations, and pseudocode for the factorization objective. It describes the VAE architecture and training settings, dataset-specific factor correlations, and the pairwise support-distance computation.

H FURTHER EXPERIMENTAL DETAILS

  • Experiments used PyTorch and reimplemented baselines for exact and fair comparability; the VAE used 64 × 64 inputs, 10 latent dimensions, batch size 64, Adam, learning rate 10^-4, and 300,000 training steps.The encoder used convolutional layers and MLPs; the decoder used an MLP and upconvolutions.Source: H FURTHER EXPERIMENTAL DETAILS · p. 25

H.1 CORRELATION SETTINGS

  • Correlation settings enumerate paired factors, shared confounders, inverse correlations, and no-correlation baselines across the evaluated datasets.For example, the Shapes3D settings include position and object-factor pairings, while the appendix defines multiplying pairwise probabilities to introduce multiple correlated pairs.Source: §H.1 · pp. 25–26

I PSEUDOCODE

  • The pseudocode approximates pairwise Hausdorff distance by subsampling latent pairs, forming their batchwise Cartesian-product support, and summing pairwise distance contributions.It computes distances between the factorized support and the two-dimensional batch embeddings.Source: I PSEUDOCODE · p. 27
Loading 2210.07347v3…