Disentangling Factors of Variation Using Few Labels

• Unsupervised disentanglement methods vary widely, and identifying well-disentangled runs without supervision is difficult, consistent with the impossibility result without inductive biases.Source: §1 · p. 1
• A few labels explicitly encode prior knowledge, complementing implicit architectural biases whose effects may be hard to control.The authors describe collecting a small number of human annotations as simple and cheap through crowdsourcing.Source: §1 · p. 2
• The study trains more than 52,000 models and finds that supervised validation enables reliable learning, while incorporating labels into training improves disentanglement and downstream performance.Source: §1 · p. 2
• Some existing metrics that require latent labels can tune unsupervised methods with very few labeled examples, making supervised model selection a viable approach.Source: §1 · p. 2
• Adding a supervised loss with as few as 100 labeled examples outperforms unsupervised training with supervised validation on disentanglement scores and downstream performance.Both approaches also tolerate noisy and coarse labels and partial annotations.Source: §1 · p. 3

• Disentanglement seeks a representation whose independent components separate ground-truth factors, so changing one latent factor corresponds to changing one representation dimension.Source: §2 · p. 3
• BetaVAE and FactorVAE scores use interventions to predict which factor was changed, whereas MIG, Modularity, DCI Disentanglement, and SAP compare factor-code relationships.Source: §2 · p. 3
• Many unsupervised methods extend VAEs with regularizers that structure the latent space, using an optimization template based on the ELBO and a regularization term.Source: §2 · p. 3
• β-VAE and AnnealedVAE reduce bottleneck capacity, Factor-VAE and β-TCVAE penalize total correlation, and DIP-VAE variants match aggregated-posterior moments to a factorized distribution.Source: §2 · p. 3
• Some semi-supervised approaches separate observed factors from other latent variables that remain entangled, while this paper aims to separate all factors of variation.The paper places this goal alongside work on disentangling pose from content, class labels, and nonlinear ICA.Source: §2 · pp. 3–4

3.1 EXPERIMENTAL SETUP AND APPROACH

• The experiments use dSprites, Cars3D, SmallNORB, and Shapes3D, with 100 or 1,000 labels and perfect, binned, noisy, or partial annotations.Source: §3.1 · p. 4
• MIG, DCI Disentanglement, and SAP are used for model selection because they work on observational data, unlike intervention-based BetaVAE and FactorVAE scores.Source: §3.1 · p. 4
• Across 32 settings, the authors train cohorts of four VAE methods and select models using metrics computed on the labeled sets before evaluating them on 10,000 test samples.Each cohort includes 36 models, covering six hyperparameters and six random seeds.Source: §3.1 · pp. 4–5

3.2 KEY FINDINGS

• MIG and DCI generally correlate well with test metrics, while SAP is weaker with 100 labels but improves with 1,000; coarse labels do not appear detrimental to model selection.The SAP classification problem becomes easier with binned labels, but observing only two factors lowers correlation relative to other label corruptions.Source: §3.2 · p. 5
• With 100 labeled examples, MIG and DCI Disentanglement identify good runs on the considered datasets; binned and noisy labels remain useful, while two observed factors still yield high but comparatively lower test-score correlation.The authors recommend labeling more factors coarsely rather than fewer factors more accurately when possible.Source: §3.2 · pp. 5–6, Figure 2 · p. 5

4 INCORPORATING LABEL INFORMATION DURING TRAINING

• The supervised regularizer constrains the VAE latent space using a loss on the few labeled observation-factor pairs, with binary cross-entropy matching normalized targets to the encoder mean.Only the first d latent dimensions are regularized when the latent space has more dimensions than the number of factors.Source: §4 · p. 6
• The approach assumes labels for all ground-truth factors on a limited number of observations, unlike prior semi-supervised work that separates observed factors from entangled ones.The authors note that adapting disentanglement scores to the latter setting is unclear.Source: §4 · p. 6

4.1 EXPERIMENTAL SETUP

• The experiments span 40 settings across four data sets, two label-set sizes, and five labeling conditions; each setting includes model cohorts and supervised model selection.The semi-supervised cohorts contain 28,800 models overall; the fully supervised baseline uses cohorts of six models per labeled data set.Source: §4.1 · pp. 6–7

4.2 SHOULD LABELS BE USED FOR TRAINING?

• For semi-supervised training, stronger regularization makes pairwise mutual information between factors and representations closer to diagonal, unlike the reported unsupervised comparison.The paper presents this diagonal structure as a proxy for improved disentanglement.Source: §4.2 · p. 7
• With 100 labeled examples, S2/S wins 70.5% of trials over the compared approaches on a random disentanglement metric and data set.Figure 4 compares the probability that each method is best across a random test metric and data set.Source: §4.2 · p. 7, Figure 4 · p. 8
• S2/S methods often outperform U/S methods on downstream prediction of ground-truth factors from latent representations.The evaluation uses training-set sizes of 10, 100, 1000, and 10,000 examples.Source: §4.2 · p. 8
• The authors conclude that semi-supervised training can outperform supervised model selection, with a larger benefit when more labels are available, while sample-efficient disentanglement metrics remain a practical research need.The fully supervised baseline performs worse than methods that also use unsupervised data.Source: §4.2 · pp. 7–8

4.3 HOW ROBUST IS SEMI-SUPERVISED TRAINING TO IMPRECISE LABELS?

• Imprecise labels do not significantly worsen either approach; with 100 labels, S2/S beats U/S in 64.8% of binned-label and 67.5% of noisy-label trials.For partial labels, the approaches are equivalent at 100 examples (50.0%); S2/S improves to 62.6% at 1000 examples.Source: §4.3 · p. 8, Figure 5 · p. 8

5 CONCLUSION

• Existing disentanglement metrics can select unsupervised models with very few labels, including noisy labels.Adding labels through a simple supervised regularizer can yield even better results.Source: §5 · p. 8
• Both unsupervised model selection and semi-supervised training are robust to imprecise labels and partial factor labeling.The authors describe these approaches as readily applicable to real-world machine-learning systems.Source: §5 · pp. 8–9
• The findings provide practical guidelines for developing machine-learning systems using limited factor labels.The authors hope the findings will help advance disentanglement research toward more practical data sets and tasks.Source: §5 · p. 9

A ORDERING AS AN INDUCTIVE BIAS

• Removing label ordering significantly harms S2/S test disentanglement scores, even though the test metrics are invariant to permutations.The study permutes factor-value order to remove the ordinal information.Source: A ORDERING AS AN INDUCTIVE BIAS · p. 13
• With permuted labels, U/S is superior to the corresponding S2/S methods most of the time, though the gap reduces with more labels.The comparison uses a random disentanglement metric and data set with perfect labels for U/S.Source: A ORDERING AS AN INDUCTIVE BIAS · p. 13
• A supervised regularizer using ordinal structure is useful when its assumptions are correct; when they are incorrect, it offers no benefit over U/S model selection.U/S model selection is invariant to ordinal structure.Source: A ORDERING AS AN INDUCTIVE BIAS · p. 13

B ARCHITECTURES AND DETAILED EXPERIMENTAL DESIGN

• The main experiment uses the default disentanglement-lib architecture, with encoder and decoder details listed in Table 2.The discriminator for FactorVAE and other fixed hyperparameters are documented in Tables 4b and 4c.Source: B ARCHITECTURES AND DETAILED… · p. 13, Table 2 · p. 14
• The methods use explored hyperparameters listed in Table 3, while reported single-valued hyperparameters are fixed based on prior literature.Other fixed hyperparameters are listed in Table 4.Source: B ARCHITECTURES AND DETAILED… · p. 13, Table 3 · p. 15, Table 4 · p. 15
• Figure 6 compares unsupervised and semi-supervised models with perfect labels against semi-supervised training with permuted labels.The comparison examines the inductive bias provided by label ordering.Source: Figure 6 · p. 14

C DETAILED PLOTS FOR SECTION 3

• Rank correlation between validation and test metrics improves when validation uses 1000 rather than 100 examples.Figures 7 and 8 report these correlations across data sets.Source: C DETAILED PLOTS FOR SECTION 3 · p. 15
• Figures 10–15 show rank correlations between validation metrics computed with binned, noisy, or partial observations and test metrics with exact labels, for 100 and 1,000 examples across all data sets.Source: C DETAILED PLOTS FOR SECTION 3 · p. 15
• Figures 10–15 show rank correlations between validation metrics computed with binned, noisy, or partial observations and test metrics computed with exact labels, using 100 or 1000 examples across all datasets.Source: C DETAILED PLOTS FOR SECTION 3 · p. 15
• Validation MIG computed from 100 labeled examples has a strong linear relationship with test DCI on dSprites.Figure 9 plots these values for each unsupervised model.Source: C DETAILED PLOTS FOR SECTION 3 · p. 15

D.1 DOES SUPERVISION HELP TRAINING?

• Figure 18 compares median test scores after U/S validation with 100 or 1000 labeled examples across data sets.U/S models are validated with MIG, which has higher rank correlation with most test metrics than other validation metrics.Source: §D.1 · p. 15
• S2/S often outperforms the corresponding U/S method on disentanglement metrics, especially with more labels.Table 5 reports how often each S2/S method outperforms its corresponding U/S method.Source: §D.1 · p. 18
• With 1000 labeled examples, S2/S is often better than corresponding U/S methods even when U/S validation uses test MIG with 10000 examples.The authors report similar trends when U/S validation uses test DCI Disentanglement.Source: §D.1 · p. 18
• Semi-supervised methods often outperform U/S methods in downstream performance, particularly when more labels are available.Table 6 reports how often S2/S improves downstream performance over U/S.Source: §D.1 · p. 18

D.2 WHAT HAPPENS IF WE COLLECT IMPRECISE LABELS?

• Imprecise labels do not significantly worsen either supervised validation or semi-supervised training performance.Figures 21 and 22 extend the comparison across sample sizes, test scores, and data sets.Source: §D.2 · p. 18, Figure 21 · p. 22, Figure 22 · p. 23
• Increasing supervised regularization makes the mutual-information matrix between z and r(x) closer to diagonal.Figure 19 presents this relationship.Source: Figure 19 · p. 21
• Figure 20 compares median downstream performance after validation with 100 versus 1000 examples.Its downstream tasks are cross-validated Logistic Regression and Gradient Boosting with training sets of 10, 100, 1000, or 10000 examples.Source: Figure 20 · p. 22
• Figures 21 and 22 compare perfect, binned, noisy, and partial labels for U/S and S2/S at 100 and 1000 samples.Tables 7–9 report how often S2/S outperforms U/S with binned, noisy, or partial labels.Source: Table 7 · p. 21, Figure 21 · p. 22, Figure 22 · p. 23