Flexibly Fair Representation Learning by Disentanglement

Elliot Creager; David Madras; Jörn-Henrik Jacobsen; Marissa A. Weis; Kevin Swersky; Toniann Pitassi; Richard Zemel

Source-linked AI summary

Flexibly Fair Representation Learning by Disentanglement

Elliot Creager, David Madras, Jörn-Henrik Jacobsen, Marissa A. Weis, Kevin Swersky, Toniann Pitassi, Richard Zemel

arXiv:1906.02589v1Jun 6, 2019cs.LGcs.AIstat.ML

TL;DR

The paper addresses fair representation learning when fairness must cover multiple sensitive attributes and their conjunctions without fixing the sensitive attributes at training time. It proposes FFVAE, which disentangles sensitive information into manipulable latent subspaces and supports test-time fairness adaptation without sensitive attributes. Empirically, FFVAE matches or exceeds baseline fairness–accuracy tradeoffs on most evaluated subgroups, while broader fairness metrics and distributional shift remain future challenges.

Problem
Existing fair representation methods are flexible across downstream labels but generally require the sensitive attribute to be specified during training, limiting multi-attribute and subgroup fairness.
Method
FFVAE uses multiple sensitive attributes as labels to induce a factorized latent structure whose sensitive subspaces can be removed or composed at test time.
Results
FFVAE matches or exceeds existing disentangled representation approaches on the fairness–accuracy tradeoff for a majority of evaluated subgroups.
Takeaways & Limitations
A single learned representation can support fair classification with multiple sensitive attributes and their conjunctions even when test-time sensitive labels are unavailable.
Takeaways & Limitations
Future work must address fairness metrics beyond demographic parity and robustness to distributional shift.

Abstract

from arXiv · show

We consider the problem of learning representations that achieve group and subgroup fairness with respect to multiple sensitive attributes. Taking inspiration from the disentangled representation learning literature, we propose an algorithm for learning compact representations of datasets that are useful for reconstruction and prediction, but are also \emph{flexibly fair}, meaning they can be easily modified at test time to achieve subgroup demographic parity with respect to multiple sensitive attributes and their conjunctions. We show empirically that the resulting encoder---which does not require the sensitive attributes for inference---enables the adaptation of a single representation to a variety of fair classification tasks with new target labels and subgroup definitions.

1. Introduction

The paper targets fair representations that remain adaptable to multiple sensitive attributes and subgroups at test time. FFVAE uses disentanglement to support composable fairness modifications without requiring sensitive attributes during inference, and experiments show favorable fairness–accuracy tradeoffs.

Existing fair representations adapt to different prediction labels but typically require one sensitive attribute to be fixed during training.Mis-specified or overly constraining sensitive attributes may hurt downstream prediction performance.
FFVAE uses multiple sensitive labels during training to disentangle their influences into a representation that can be modified at test time.The encoder does not require sensitive attributes for inference.
The method is evaluated on correlated synthetic data, Communities & Crime, and Celeb-A for downstream fair classification.The synthetic setting tests disentanglement when factors are correlated, while the real datasets extend evaluation across tabular and image data.
On a majority of evaluated subgroups, FFVAE matches or exceeds the fairness-accuracy tradeoff of existing disentangled representation approaches.

2. Background

Fair representation learning seeks compact encodings that remain useful for downstream prediction while supporting fairness across multiple sensitive attributes and their combinations. The background connects demographic parity and its distance metric to disentangled VAE representations.

Demographic parity requires predictions to be independent of sensitive attributes, while demographic parity distance measures deviation from parity.∆DP = 0 indicates demographic parity, and lower ∆DP indicates less unfairness.
Multi-attribute fair classification methods address subgroup fairness directly, whereas this paper focuses on representations that support multiple downstream fair classification tasks.
Representation learning aims to produce an encoding from which predictors inherit independence from sensitive attributes.If z is independent of a, a predictor derived from z is intended to satisfy demographic parity with respect to a.
This work extends fair representation learning from single attributes to vectors of attributes, requiring fairness for each dimension and their combinations.
VAE variants encourage disentanglement through stronger KL weighting or aggregate-posterior factorization.β-VAE increases the KL weight, while FactorVAE and β-TCVAE target factorization of latent components.
Disentangled latent codes assign semantically meaningful variation to separate dimensions, allowing factors to be manipulated in isolation.

3. Related Work

Prior work studies subgroup fairness, disentangled representations, and fair representation learning, but the paper positions its method as the first multi-attribute fair representation learning approach.

Multi-attribute fair classification has used statistical parity, calibration, or multi-group accuracy procedures for identifiable subgroup classes.
Disentanglement research seeks independent latent components that explain observed data and has expanded substantially with neural representation learning.
The paper claims the first multi-attribute fair representation learning method, using sensitive attributes as labels to induce a factorized aggregate latent code.

4. Flexibly Fair VAE

FFVAE structures representations into task-useful and sensitive subspaces so sensitive influences can be removed compositionally after training. Its objective combines reconstruction, sensitive-attribute predictiveness, disentanglement, and prior matching, while inference requires no sensitive attributes.

Method: FFVAE isolates each sensitive attribute in a specific latent subspace and factorizes those subspaces independently.
Desiderata: FFVAE’s latent code is designed to be both disentangled from sensitive subspaces and predictive of corresponding sensitive attributes.
Scope: The method accommodates binary sensitive attributes with matching sensitive dimensions and can also accommodate categorical or continuous attributes.
Flexible fairness: The representation can achieve demographic parity for an attribute by removing or replacing its corresponding sensitive dimension.For conjunctions of binary attributes, the corresponding multiple sensitive dimensions are removed together.
Model: The factorized decoder reconstructs non-sensitive observations from z and predicts sensitive attributes from b.The sensitive-attribute decoder uses a factorized Bernoulli classifier with each bj as the logit for aj.
Objective: The learning objective combines reconstruction, predictiveness, disentanglement, and dimension-wise KL terms.α controls sensitive-attribute alignment, while γ controls aggregate independence in the latent code.
Inference: The model requires sensitive attributes during training but not at test time.This addresses settings where sensitive information is difficult to collect because of practical or legal restrictions.

5. Experiments

The experiments evaluate FFVAE through held-out auditing tasks for fair classification, predictiveness, and disentanglement across synthetic, tabular, and image datasets. FFVAE generally improves fairness-accuracy tradeoffs and supports fair classification across multiple sensitive subgroups.

Evaluation procedure: The auditing pipeline splits data, trains an encoder, freezes it, and evaluates MLP predictors on held-out data.Fair-classification audits modify encoder outputs before predicting held-out labels.
DSpritesUnfair: On DSpritesUnfair, FFVAE nearly eliminates demographic-parity gaps when sensitive attributes correlate with the target, with little accuracy loss.Baselines improve ∆DP by no more than about 0.05 in this setting.
DSpritesUnfair: For conjunctions of sensitive attributes on DSpritesUnfair, FFVAE matches or exceeds baseline accuracy at a given ∆DP almost everywhere.The encoder training uses Shape and Scale, while downstream prediction targets XPosition.
Disentanglement and predictiveness: Increasing α improves both FFVAE predictiveness and disentanglement in audits aggregated across models from the correlated DSprites setting.The disentanglement audit makes predicting the removed sensitive attribute more difficult as α increases.
Communities & Crime: On Communities & Crime, FFVAE achieves higher accuracy than baselines in general despite correlated sensitive attributes, though it does not outperform baselines on every conjunction.The dataset uses racePctBlack, blackPerCapIncome, and pctNotSpeakEnglWell as sensitive attributes and violent crimes per capita as the held-out label.
Celeb-A: On Celeb-A, FFVAE provides favorable fairness-accuracy tradeoffs across many subgroups, but neither method dominates and some subgroups lack low-∆DP solutions.For example, FFVAE is better across all ∆DP levels for ¬E∧M, but does not find a low-∆DP solution for ¬C∧¬E.

6. Discussion

FFVAE learns structured representations that can be adapted compositionally at test time for fairness across multiple sensitive attributes and their conjunctions. Real-data performance is promising but not robust across all subgroups, underscoring the difficulty of subgroup-fair representation learning.

FFVAE modifies representations at test time for fairness across multiple sensitive attributes and their conjunctions, even without test-time sensitive-attribute labels.
On synthetic image data with correlated attributes and labels, FFVAE disentangled sensitive sources of variation.
FFVAE achieved better parity for a given accuracy budget across several group and subgroup definitions than baseline disentanglement algorithms.
On Communities & Crime and Celeb-A, FFVAE performed well, but no model was robust across all possible real-data subgroups.
Future work includes extending flexible fairness beyond demographic parity and improving robustness to distributional shift in real data.

A. Discriminator approximation of total correlation

The FFVAE discriminator estimates dependence between latent subspaces by contrasting real encoder outputs with dimension-shuffled fake samples. Its adversarial loss is combined with the FFVAE objective to encourage independence and stable optimization.

The discriminator contrasts true samples from the aggregate posterior with fake samples drawn from products of marginal distributions.
Dimension-shuffling latent subspaces breaks their joint statistics while preserving marginal statistics, producing equal numbers of real and fake training examples.
The encoder receives the adversary’s error through backpropagation, and random-chance discrimination indicates the desired independence property.
A binary discriminator with indicator u distinguishes aggregate-posterior samples from samples drawn from the product of marginals.
The discriminator outputs a real-sample probability, while its cross-entropy training calibrates an approximation to the relevant log density ratio.
The discriminator and FFVAE objectives are optimized in a min-max fashion, with single-step alternating updates yielding stable optimization in the experiments.

B. DSpritesUnfair Generation

The DSpritesUnfair variant introduces correlation between Shape and X-position while retaining uniform sampling for other factors. Its factors are then converted into binary variables for fairness experiments.

The original DSprites dataset contains six ground-truth factors of variation, including Shape, Scale, XPosition, and YPosition.
DSpritesUnfair correlates Shape and X-position instead of factorizing their joint distribution.
The correlated Shape–X-position pair is sampled with qS = 1 and qX = 3, while all other attributes remain uniformly sampled.
The factors of variation are binarized using boolean outputs of specified operations.

C. DSprites Architectures

The DSprites implementation uses convolutional encoder and decoder networks with a latent code, plus a multilayer discriminator for FFVAE training. The encoder returns latent means and log-variances, while the discriminator returns logits and probabilities.

The architecture specification covers convolutional encoder q(z, b|x), decoder q(x|z, b), and the FFVAE discriminator.
The convolutional encoder is configured for 64×64 single-channel images with latent_dim = 10.
The encoder splits its network output into latent means and log-variances and returns both.
The convolutional decoder is configured for 64×64 images, a 10-dimensional latent code, and one channel.
The discriminator uses repeated linear layers with LeakyReLU activations and produces two output classes followed by softmax probabilities.

D. DSpritesUnfair Training Details

The experiments use Adam optimization with a 0.001 learning rate, 3×10^5 encoder iterations, and minibatches of 64; the MLP classifier has two 128-unit hidden layers.

Adam optimization used a 0.001 learning rate for all network parameters.
Encoders trained for 3×10^5 iterations with minibatch size 64.
The MLP classifier used two hidden layers with 128 units each and validation-loss patience of 5 epochs.

E. Mutual Information Gap

The Mutual Information Gap evaluates whether FFVAE latent variables align individually with known factors after training on biased DSpritesUnfair data and testing on balanced DSprites. MIG generally improves with α, while γ has a context-dependent relationship with disentanglement and training stability.

MIG measures the gap between the highest and second-highest mutual information values for each factor, rewarding one latent variable per factor.The metric is computable here because the synthetic dataset provides ground-truth factors of variation.
Models were trained on biased DSpritesUnfair data and evaluated on balanced DSprites data to assess disentanglement across distributions.
MIG increases with α across γ values, except for training instability at the highest γ values.
For α ≥300, MIG improves as γ increases, whereas the relationship is harder to assess when α is small because training becomes unstable.
For low α, increasing γ worsens MIG, contrasting with improvement at sufficiently high α and indicating a complex α–γ relationship.
Across combinations, higher γ/α generally yields lower MIG, with the highest MIG around log(γ/α) = −2.
The learned representations show evidence of being at least partially invariant to interventions on sensitive attribute a.This interpretation connects the Figure 6 results to counterfactual fairness.