Source-linked AI summary
The Variational Fair Autoencoder
Christos Louizos, Kevin Swersky, Yujia Li, Max Welling, Richard Zemel
TL;DR
The paper addresses how to learn representations that remove known sensitive or nuisance factors while retaining task-relevant information. It introduces the Variational Fair Autoencoder, combining variational autoencoding, factorized priors, and MMD regularization, and reports effective invariant representations across fair classification, domain adaptation, and general invariance tasks.
Problem
The central problem is learning latent representations that are minimally informative about sensitive or nuisance variables while remaining informative about observed targets and the data.
Method
The Variational Fair Autoencoder combines a deep variational autoencoder with factorized priors, jointly optimized semi-supervised layers, and an MMD penalty matching latent posterior moments across sensitive-variable groups.
Results
The method learns representations that remove unwanted information while retaining informative content, improving the reported accuracy–invariance tradeoff in fair classification and outperforming DANN on 9 of 12 domain-adaptation tasks.
Takeaways & Limitations
VFAE supports fair classification, domain adaptation, and invariant-representation learning by explicitly purging known sensitive or domain information from latent representations.
Abstract
from arXiv · showhide
We investigate the problem of learning representations that are invariant to certain nuisance or sensitive factors of variation in the data while retaining as much of the remaining information as possible. Our model is based on a variational autoencoding architecture with priors that encourage independence between sensitive and latent factors of variation. Any subsequent processing, such as classification, can then be performed on this purged latent representation. To remove any remaining dependencies we incorporate an additional penalty term based on the "Maximum Mean Discrepancy" (MMD) measure. We discuss how these architectures can be efficiently trained on data and show in experiments that this method is more effective than previous work in removing unwanted sources of variation while maintaining informative latent representations.
1 INTRODUCTION
The paper frames representation learning as retaining task-relevant information while removing nuisance variation, and applies this goal to purging known sensitive factors from latent representations.
- Representation learning seeks representations that are informative for a task while removing uninformative or detrimental factors of variation.Such factors are commonly described as noise or nuisance variables, while retained information corresponds to latent or hidden factors.
- The paper targets latent representations that remove nearly all information about known sensitive or nuisance factors while retaining as much information about the data as possible.The desired representation is maximally informative about an observed target y and minimally informative about a sensitive or nuisance variable s.
- The proposed approach uses a deep variational autoencoder with factorized priors and an MMD penalty to reduce residual dependencies between sensitive and latent variables.The combined method is evaluated for learning representations that remove unwanted information while retaining remaining informative content.
2 LEARNING INVARIANT REPRESENTATIONS
The paper models invariant representations by separating nuisance variation from remaining information, then uses semi-supervised structure and MMD regularization to reduce residual dependence.
- 2.1 UNSUPERVISED MODEL: The generative model separates observed nuisance variation s from continuous latent information z using independent sources.The latent representation is inferred from x and s under a model whose prior encodes marginal independence.
- 2.1 UNSUPERVISED MODEL: A variational autoencoder parameterizes both the decoder pθ(x|z, s) and encoder qφ(z|x, s) with neural networks optimized jointly.Training uses the SGVB algorithm and a lower bound on log-likelihood.
- 2.2 SEMI-SUPERVISED MODEL: When nuisance s and label y correlate, unsupervised factorization can produce degenerate label representations, motivating label injection through a second latent layer.The semi-supervised model uses z1 for invariant features and z2 for variation not explained by y, with qφ(y|z1) imputing missing labels.
- 2.2 SEMI-SUPERVISED MODEL: Joint optimization lets label information guide feature extraction while avoiding the degeneration risk of separately training stochastic layers.The authors state that this can preserve more predictive information by directing extraction toward salient data features.
- 2.3.1 MAXIMUM MEAN DISCREPANCY: Residual dependence between sensitive s and latent z1 can leak through the posterior, especially when labels and sensitive variables are correlated.The paper addresses this remaining dependence with regularization on marginal posterior distributions.
- 2.3.1 MAXIMUM MEAN DISCREPANCY: MMD regularization matches marginal posterior distributions across sensitive groups, yielding the Variational Fair Autoencoder.For a universal kernel, MMD is asymptotically zero exactly when the compared distributions are equal; random kitchen sinks provide a faster minibatch approximation, using D = 500 in experiments.
3 EXPERIMENTS
Experiments evaluate whether VFAE removes sensitive or domain information while preserving label-relevant information across fair classification, domain adaptation, and invariant-representation tasks.
- Datasets: The experiments cover three fairness datasets, Amazon reviews for domain adaptation, and Extended Yale B for invariant representation learning.The fairness datasets are German, Adult, and Health; Extended Yale B varies lighting conditions while predicting person identity.
- Evaluation: Evaluation measures sensitive-variable removal using LR and RF prediction, discrimination metrics, and label-prediction accuracy.For VFAE and VAE, label predictions use qφ(y|z1), while sensitive-variable classifiers predict s from z1.
- Fair classification: On Adult, VFAE removed more sensitive information while retaining most predictive information, and MMD considerably reduced discrimination relative to VAE.LFR achieved the highest y accuracy and lowest discrimination but retained sensitive information detectable by random forest.
- Fair classification: On German, MMD improved VFAE discrimination scores over VAE while label accuracy remained similar; on Health, VFAE and LFR retained less sensitive information than VAE.Health label prediction was dominated by the majority class because only 15% of patients were admitted to hospital.
- Domain adaptation: On Amazon reviews, sensitive-domain accuracy approached random chance of 0.5, while VFAE label accuracy exceeded DANN on 9 of 12 tasks and was similar on the other 3.The experiments used LR and RF for domain prediction and compared label performance with DANN.
- Invariant representations: On Extended Yale B, VFAE reduced sensitive lighting accuracy from 96% to about 50% and increased identity accuracy from 78% to about 85%.The resulting embeddings clustered primarily by person identity rather than lighting condition.
4 RELATED WORK
The paper contrasts VFAE’s fair-representation approach with prior clustering and domain-adaptation methods, emphasizing distributed representations and higher-order distribution matching.
- Zemel et al.’s method learns localized cluster assignments with approximately equal group proportions across sensitive groups.
- That clustering restriction cannot leverage the representational power of distributed representations.
- The earlier penalty can fail to account for higher-order moments, allowing sensitive information to leak despite satisfying the penalty.
- Figure 5 compares t-SNE visualizations of the Extended Yale B training set in the original x space and VFAE’s latent z1 space.
- VFAE addresses these issues, while related domain-adaptation methods seek invariant features by matching distributions across domains.
5 CONCLUSION
The paper presents VFAE as a model for invariant representations, evaluates it on fair classification, domain adaptation, and general representation learning, and reports improvements or competitiveness across these applications.
- VFAE extends a semi-supervised VAE to learn representations invariant to known dataset aspects while retaining remaining information.
- An MMD regularizer further promotes invariance in the posterior distribution over latent variables.
- For fair classifiers, VFAE produces a better tradeoff between accuracy and invariance.
- For domain adaptation, VFAE is competitive with recently proposed adversarial approaches.
- For general invariant representation learning, VFAE clearly improves over a neural network incorporating an MMD penalty.
A DISCRIMINATION METRICS
This appendix formalizes discrimination metrics that evaluate predictions across groups defined by the nuisance variable and, for one metric, incorporate predicted class probabilities.
- The discrimination metric and its probability-aware variant are mathematically formalized for predictions grouped by nuisance-variable values.
- Ns=0 denotes the number of datapoints with nuisance variable s = 0.
- For predictions and their corresponding probabilities, the experiments use a Logistic Regression classifier.
B PROXY A-DISTANCE (PAD) FOR AMAZON REVIEWS DATASET
The paper measures source–target domain distinguishability with Proxy A-distance, comparing VFAE’s representations with DANN on Amazon reviews.
- Proxy A-distance approximates the H-divergence measure of source–target domain distinguishability.
- It is computed by training a classifier to discriminate source from target examples and applying the resulting test error ϵ.
- Low PAD scores indicate that the classifier poorly discriminates between source and target domain examples.
- VFAE’s PAD scores are overall lower than DANN’s, indicating better factorization of domain information in the new representation.
- Figure 6 places VFAE results on the left and DANN results on the right for the Amazon reviews dataset.