Source-linked AI summary
Learning Deep Disentangled Embeddings with the F-Statistic Loss
Karl Ridgeway, Michael C. Mozer
TL;DR
Deep embeddings expose class structure while disentangling methods expose independent factors, but the two goals have been only weakly connected. The paper introduces an F-statistic loss that separates classes on selected embedding dimensions, matching strong identity-classification performance and outperforming β-VAE variants on reported disentanglement measures. The main limitation is the loss’s equal-variance assumption, although the authors report robustness when class sizes are similar.
Problem
Deep embeddings target class structure and few-shot learning, whereas disentangling targets multiple independent factors; the paper addresses the limited connection between these goals.
Method
The paper uses an F-statistic loss that maximizes class-pair separation on selected embedding dimensions and trains with either class-aware or unnamed-factor supervision.
Results
The method matches state-of-the-art recall@1 performance and achieves the best modularity on both datasets, the best explicitness on small NORB, and higher modularity and explicitness than β-VAE variants.
Takeaways & Limitations
Selective dimension-wise separation can support class embeddings while producing representations evaluated as more modular and explicit than the compared alternatives.
Takeaways & Limitations
The loss assumes equal class variances, although its likelihood is fairly robust when class sizes are approximately equal.
Abstract
from arXiv · showhide
Deep-embedding methods aim to discover representations of a domain that make explicit the domain's class structure and thereby support few-shot learning. Disentangling methods aim to make explicit compositional or factorial structure. We combine these two active but independent lines of research and propose a new paradigm suitable for both goals. We propose and evaluate a novel loss function based on the $F$ statistic, which describes the separation of two or more distributions. By ensuring that distinct classes are well separated on a subset of embedding dimensions, we obtain embeddings that are useful for few-shot learning. By not requiring separation on all dimensions, we encourage the discovery of disentangled representations. Our embedding method matches or beats state-of-the-art, as evaluated by performance on recall@$k$ and few-shot learning tasks. Our method also obtains performance superior to a variety of alternatives on disentangling, as evaluated by two key properties of a disentangled representation: modularity and explicitness. The goal of our work is to obtain more interpretable, manipulable, and generalizable deep representations of concepts and categories.
1 Using the F statistic to separate classes
The F-statistic loss measures class separation statistically and applies that measure selectively across embedding dimensions. This design provides a probability-based stopping criterion while encouraging dimensions to align with disentangled factors.
- The method quantifies class separation using an ANOVA F statistic, the ratio of between-class to within-class variability.
- Under equal-mean and normality assumptions, large F statistics indicate class separation unlikely under the null distribution.The statistic follows a Fisher-Snedecor distribution with degrees of freedom 1 and n1 + n2 − 2.
- The likelihood is fairly robust to unequal variances when class sizes are approximately equal, but the approach restricts the F statistic to class pairs.The general multiclass likelihood is considered a weak indicator of class separation.
- 1.1 From one to many dimensions: For multidimensional embeddings, the loss selects the d dimensions with the largest separation measures and ignores the others.The selected dimensions are those where the class pair is already best separated.
- 1.1 From one to many dimensions: Optimizing only a separated subset allows remaining dimensions to preserve semantic similarities while supporting disentangled features.The loss is not rotation-invariant because it focuses separation on specific dimensions.
- 1.1 From one to many dimensions: The loss gradient drops to zero after reliable separation on at least d dimensions, creating a natural stopping criterion.The achieved separation depends on the number of samples per class.
2 Identity classification
The paper evaluates the F-statistic loss for identity matching on person and bird image datasets using recall@1. It matches the strongest competing losses overall, with the best result on CUHK03, a tie on Market-1501, and a lower tier on CUB-200.
- The identity-classification evaluation covers CUHK03, Market-1501, and CUB-200-2011 using person and bird image-matching tasks.
- Recall@1 classifies each query by whether its nearest held-out reference belongs to the same class.The evaluation is equivalent to q-shot learning, with q ranging from 3 to 10 for these datasets.
- 2.2 Results: The F-statistic loss achieves accuracy comparable to the best competitors, histogram and binomial deviance losses.
- 2.2 Results: It obtains the best result on CUHK03, ties on Market-1501, and is a tier below the best on CUB-200.
3 Quantifying disentanglement
The paper evaluates disentanglement through modularity and explicitness, emphasizing that dimensions should isolate individual factors while the code remains useful for recovering factor values. It formalizes these criteria using mutual information and classifier-based recovery.
- Disentanglement criteria: A modular code dimension conveys information about at most one factor, while explicitness measures whether factor values can be recovered from the code.The paper treats modularity and explicitness as defining criteria, while rejecting compactness as necessary.
- Modularity: The evaluation quantifies modularity by comparing each code dimension’s mutual-information profile with an ideal single-factor template.For dimension i and factor f, mutual information is denoted m_if, and the template selects the factor with maximum mutual information.
- Modularity: A deviation of 0 indicates perfect modularity, whereas 1 indicates equal mutual information with every factor; modularity is computed as the mean of 1 − δ_i.The overall score averages the per-dimension modularity scores.
- Explicitness: Explicitness measures code coverage by testing whether discrete factor values can be recovered with a simple classifier.The authors use one-versus-rest logistic regression because its implied linear separability was more robust than RBF networks in their experiments.
- Evaluation: The paper evaluates its modularity and explicitness quantification against other disentangling and deep-embedding methods.These measures are used as the basis for the following empirical comparison.
4 A weakly supervised approach to disentanglement
The paper introduces unnamed-factor oracle supervision as an intermediate alternative between unsupervised and factor-aware disentangling. On Sprites and small NORB, the F-statistic loss achieves the best modularity on both datasets and the best explicitness on small NORB, while β-VAE exhibits a modularity–explicitness trade-off.
- Motivation: Unsupervised disentangling is underconstrained, whereas factor-aware methods require oracles that identify or organize instances by a named factor.The proposed setting seeks an intermediate degree of supervision.
- Unnamed-factor supervision: An unnamed-factor oracle groups or orders instances by similarity along a chosen factor without revealing the factor’s name.The authors argue that such judgments can be obtained through crowdsourcing and generalize class-similarity training.
- Datasets: The experiments use Sprites with seven identity factors and small NORB with toy type, elevation, and azimuth as factors, treating lighting as noise.Sprites contains 672 identities formed by seven factors; small NORB contains 50 toy types across viewpoints and lighting conditions.
- Experimental design: The evaluation uses five-fold cross validation with factor-conjunction splits, validation-based stopping by mean factor explicitness, and test results from four splits.The study compares the F-statistic loss with triplet and histogram losses in these experiments.
- Results: The F-statistic loss achieves the best modularity on both datasets and the best explicitness on small NORB, while all methods achieve good explicitness on Sprites.Figure 3 reports means across validation splits and embedding dimensions for modularity, and across splits and factor-values for explicitness.
- Comparison with β-VAE: As β increases in β-VAE, modularity improves but explicitness worsens, revealing a trade-off between the two measures.The paper identifies this previously unreported trade-off as a limitation of the β-VAE approach.
- Comparison with β-VAE: Both F-statistic representations outperform all β-VAE variants, although the comparison is not exactly fair because β-VAE is unsupervised and F-statistic training is weakly supervised.The class-aware oracle provides weaker supervision than the unnamed-factor oracle.
5 Discussion and future work
The F-statistic loss is presented as a way to unify deep embeddings and disentangling, with strong reported performance in both settings. Future work centers on adapting the dimension-selection hyperparameter d.
- The F-statistic loss is motivated by unifying deep-embedding and disentangling research.
- The method achieves state-of-the-art recall@1 with a class-aware oracle and state-of-the-art disentangling with an unnamed-factor oracle.
- The authors identify adaptive estimation of d as future work because d currently governs loss behavior for all class pairs and must be tuned per dataset.
A Desiderata for an Embedding
An embedding should separate classes while preserving useful inter-class similarity. This balance supports categorization and learning novel classes from few labeled examples.
- An embedding places same-class instances close together and different-class instances far apart to expose class structure.
- Separated clusters facilitate categorizing unlabeled instances by proximity to clusters and learning new classes from a few labeled examples.
- Figure 5 illustrates a trade-off between increasing intra-class variability, conspicuous inter-class structure, and preserving inter-class similarity.
B Criteria for Disentangling
The paper evaluates disentangled codes using modularity, compactness, and explicitness. These criteria describe factor-specific dimensions, concentrated factor information, and recoverability under simple operations.
- Modularity: Modularity requires each code dimension to convey information about at most one factor, with factor-associated partitions invariant to other factors.
- Compactness: Compactness requires each factor to be represented by only one or a few code dimensions.
- Illustrative code spaces: Figure 6 contrasts sixteen code spaces for instances varying on two factors or one factor, using shape and color to mark factor values.
- Criterion summary: Table 1 summarizes whether the sixteen Figure 6 codes satisfy modularity, compactness, and explicitness using check marks.
- Explicitness: Explicitness concerns how precisely factor values can be recovered from a code, with linear separability used as the criterion in these examples.