Fair Contrastive Learning for Facial Attribute Classification

1. Introduction

• The paper frames fairness in contrastive learning as an underexplored ethical concern, defining it as outputs that are not discriminatory by sensitive attributes.It identifies ethnicity, gender, and age as examples of sensitive attributes.Source: §1 · p. 1
• SupCon can reduce its loss by learning sensitive-attribute information on biased data, although the desired representation should learn target-class information exclusively.The paper links this mechanism to unfair classification because the classifier may use sensitive attributes to set decision boundaries.Source: §1 · p. 1
• FSCL limits negative samples to those sharing the anchor’s sensitive attribute, so learning that attribute increases rather than decreases the loss.It otherwise follows supervised contrastive learning by drawing same-class samples closer than different-class samples.Source: §1 · p. 1
• Group-wise normalization balances loss contributions by group anchor and positive-sample counts to reduce disparities in intra-group compactness and inter-class separability.The paper reports that it further improves fairness with little damage to classification performance.Source: §1 · p. 2
• On CelebA and UTK Face, the method achieves the best trade-off between top-1 accuracy and fairness among the compared methods.It is also reported to be robust to data-bias intensity and effective in incompletely supervised settings.Source: §1 · p. 2

2.1. Fair Representation Learning

• Fair representation approaches described here include adversarially removing sensitive information, disentangling representation, fair knowledge distillation, and perturbing latent spaces to generate balanced data.The cited methods include different mechanisms, including gradient reversal, mutual-information minimization, and a subspace for information shared by target and sensitive attributes.Source: §2.1 · p. 2

2.2. Contrastive Representation Learning

• Contrastive learning draws same-image views together in self-supervised methods, while SupCon instead treats all samples sharing the anchor’s target class as positives.Prior methods also use memory banks or a dynamic memory queue to increase negative samples; SimCLR uses neither a memory bank nor specialized architecture.Source: §2.2 · p. 2

3. Method

• FSCL and group-wise normalization address unfairness from sensitive-attribute learning and group imbalance, respectively.The method presents a general fair representation-learning loss rather than a new contrastive-learning framework.Source: §3 · p. 3, §3.3 · p. 4, §3.4 · p. 5

3.1.1 Overall flow

• The encoder and projection network jointly learn representations from two image views, then the frozen encoder feeds a classifier trained with cross-entropy.The projection network is discarded after representation learning.Source: §3.1.1 · p. 3

3.1.2 Self-supervised and supervised contrastive losses

• Supervised contrastive loss treats other patches with the anchor’s target label as positives and normalizes across multiple positives, unlike self-supervised loss, which uses another view of the same image.The passage attributes SupCon’s stronger classification performance than the self-supervised loss to multiple positives and their normalization.Source: §3.1.2 · p. 3

3.2.1 Learning of sensitive attribute information

• Under the stated ideally biased-dataset assumption, learning sensitive-attribute information decreases LSup, so minimizing it can encourage both target and sensitive information.The analysis defines sensitive-attribute learning as increasing mutual information between representation Z and sensitive attribute S.Source: §3.2.1 · p. 3

3.2.2 Data imbalance between demographic groups

• Because anchor and positive-sample counts are not normalized within demographic groups, LSup weights majority groups more, favoring their compactness and inter-class separability.The resulting disparities between groups are linked in the passage to unfair classification performance.Source: §3.2.2 · p. 4

3.3. Fair Supervised Contrastive Loss

• FSCL uses negatives with the anchor’s sensitive attribute and a different target class, encouraging same-target similarities across groups while discouraging sensitive-attribute information.The loss encourages both intra-group and sensitive inter-group similarity to exceed target inter-group similarity.Source: §3.3 · pp. 4–5

3.4. Group-wise Normalization

• Group-wise normalization normalizes anchor and positive-sample counts within each group and experimentally mitigates group disparities in compactness and inter-class separability.It is introduced to address the disparities associated with demographic-group imbalance.Source: §3.4 · p. 5

4.4. Classification Results on CelebA

• On CelebA, FSCL+ substantially outperforms state-of-the-art fairness methods in the top-1 accuracy–EO trade-off across settings, while group-wise normalization improves fairness and preserves or improves performance.FSCL and FSCL+ significantly improve EO over Cross-Entropy and SupCon while retaining competitive accuracy.Source: §4.4 · p. 6

4.5. Adversarial Training in Contrastive Learning

• On CelebA, SupCon with GRL improves fairness somewhat but largely damages classification performance, whereas FSCL+ achieves better EO and top-1 accuracy than SupCon with GRL.Adding GRL to FSCL+ yields no further EO improvement.Source: §4.5 · p. 6

4.6. Compatibility with Fair Data Augmentation

• On CelebA, FSCL+ outperforms fair data augmentation on both EO and top-1 accuracy, and that augmentation further enhances FSCL+ fairness.The results support compatibility between FSCL+ and fair data augmentation.Source: §4.6 · p. 7

4.7. Effectiveness of Group-wise Normalization

• Group-wise normalization reduces disparities in intra-group and inter-class similarities; their standard deviations fall from 0.084 and 0.031 with FSCL to 0.038 and 0.024 with FSCL+.Similarities were evaluated across eight groups defined by attractiveness, male, and young, then normalized to sum to unity.Source: §4.7 · p. 7, Figure 2 · p. 7

4.8. Qualitative Analysis with t-SNE Visualization

• In CelebA t-SNE plots, SupCon representations separate by target class and sensitive attribute, while FSCL+ representations separate only by target class.The authors report that minority-group representations under SupCon are more similar to the counterpart class than majority-group representations.Source: §4.8 · p. 7, Figure 3 · p. 7

4.9. Robustness to Severity of Data Bias

• On UTK Face, FSCL methods best preserve fairness across imbalance intensities while maintaining top-1 accuracy close to SupCon; their EO advantage grows as α increases.The tested target and sensitive attributes are gender and ethnicity; larger α means a more imbalanced training set.Source: §4.9 · p. 7, Figure 4 · p. 8

4.10. Results in Incomplete Supervised Settings

• Without target labels during representation learning, FSCL† significantly improves fairness over SimCLR and SimCLR+GRL at an acceptable top-1 accuracy cost; with incomplete sensitive labels, FSCL+ improves EO over SupCon.FSCL+ using only 5% sensitive-attribute labels outperforms SupCon+GRL using all labels; the authors test pseudo-labeling and applying FSCL+ only to labeled data.Source: §4.10 · p. 8

4.11. Extensibility to General Bias Mitigation

• On Dogs and Cats with color bias, FSCL and FSCL+ best eliminate the bias; the authors present this as evidence of applicability beyond demographic fairness.The evaluation uses species as the target attribute and color as the sensitive attribute, with a balanced test set.Source: §4.11 · p. 8, Figure 5 · p. 8

• The paper analyzes causative factors of unfairness in supervised contrastive loss.Source: §5 · p. 8
• It proposes fair supervised contrastive loss and introduces group-wise normalization into the loss.Source: §5 · p. 8
• Experiments validate that the loss improves fairness with little degradation of classification performance.Source: §5 · p. 8

• The dataset contains m target and m sensitive classes, with equal numbers of data in each class.Source: A. Definition of Ideally Biased Dataset · p. 12
• Each target class maps to one distinct biased sensitive class, which has r times more data than the other sensitive classes within that target class.Source: A. Definition of Ideally Biased Dataset · p. 12
• The bias is strong, with r ≥ m^2, and Figure 6 depicts the setting using C for the number of data in non-biased classes.Source: A. Definition of Ideally Biased Dataset · p. 12, Figure 6 · p. 12

Assumption 1

• The proof assumes target classes are highly correlated with sensitive attributes, with r ≥ m^2.Source: Assumption 1 · p. 12

Definition 1

• Sensitive-attribute information in representation Z is defined as mutual information I(Z; S̃).Source: Definition 1 · p. 12

Conjecture 1

• As sensitive information increases, similarity to samples sharing the anchor's sensitive attribute increases, while similarity to the comparison samples decreases.Source: Conjecture 1 · p. 13

Lemma 1

• The anchor-term quantity V_a at the earlier training point is at least its value at the later point.The proof establishes this inequality for all points satisfying its stated assumptions.Source: Lemma 1 · pp. 13–14

Lemma 2

• The positive-sample term V_p is greater at the later point, while the combined quantity V is greater at the earlier point.The proof states these inequalities under its assumptions, including m > 2 and r > m^2.Source: Lemma 2 · pp. 14–15

Corollary 1

• Learning sensitive-attribute information decreases LSup, given the target and sensitive labels in the proof setting.The proof derives this from changes in the anchor and positive-sample terms.Source: Corollary 1 · p. 15, Lemma 2 · p. 15

C. Fairness Strategy in Classifier Training Stage

• Applying GRL during classifier training degrades EO and classification accuracy for FSCL+, so the added classifier-stage fairness strategy is not effective for this method.The authors speculate that retraining the encoder and projection networks with cross-entropy loss and GRL biases the fair representation.Source: C. Fairness Strategy in Classifier… · p. 15

D. Modification for Incomplete Supervised Setting

• FSCL† adapts the method to settings without target labels by using another patch from the same image as a positive and specified samples as negatives.Source: D. Modification for Incomplete… · p. 15

E. Details of t-SNE Visualization

• Across the reported t-SNE settings, FSCL+ representations are more agnostic to the sensitive attribute than SupCon representations.The plots without PCA reduction are provided in Figure 7 because PCA considerably affects representation structure.Source: E. Details of t-SNE Visualization · p. 16, Figure 7 · p. 16
• Figure 8 presents UTK Face classification accuracy and equalized odds across values of α, with gender as the target and age as the sensitive attribute.Source: Figure 8 · p. 16

• Across all α values, FSCL+ maintains the fairest EO on UTK Face with age as the sensitive attribute.Source: F. Further Experiments on UTK Face · p. 16
• Across all α values, FSCL+ also maintains the best top-1 accuracy on UTK Face with age as the sensitive attribute.Source: F. Further Experiments on UTK Face · p. 16
• With similar EO, FSCL+ has significantly higher accuracy than FD-VAE, yielding a better fairness–accuracy trade-off.Source: F. Further Experiments on UTK Face · p. 16

G. Additional Experimental Results on CelebA

• FSCL+ achieves the best accuracy–fairness trade-off across the reported CelebA results.Figure 9 visualizes the ACC–EO trade-off, with the upper-left corner indicating the optimal trade-off.Source: G. Additional Experimental Results on… · p. 16

H.1. CelebA and UTK Face

• The CelebA and UTK Face training-set compositions are detailed in Tables 8 and 9, respectively.Source: §H.1 · p. 16

H.2. Dogs and Cats

• The Dogs and Cats training set contains 3,425 black cats and white dogs and 685 white cats and black dogs, with a balanced 2,400-image test set.Unlike the cited original setting, target and bias labels are not completely correlated in training, making the task more suitable for image-level fairness methods.Source: §H.2 · p. 16

H.3. Discussion on License and Data Collection

• CelebA and UTK Face have custom non-commercial licenses, and their images include web-sourced celebrity or search-engine collections.Source: §H.3 · p. 16

I.1. Structure of Comparable Models

• The comparison models use ResNet-18 backbones, with model-specific architectures and training procedures described for Cross-Entropy, GRL, LNL, FD-VAE, MFD, SupCon, SimCLR, and FSCL.Source: §I.1 · pp. 17–18

I.2. Augmentation Strategy and Experimental Setup

• Contrastive-loss models use two augmented patches per image, and all models share the stated augmentation strategy, SGD settings, batch size, and learning rate.The augmentation sequence includes random cropping and resizing, horizontal flipping, color jittering, and grayscale conversion.Source: §I.2 · p. 18

J. Comparison with GDRO

• The authors report that their method achieves a better trade-off than GDRO on CelebA.Source: J. Comparison with GDRO · p. 18, Table 10 · p. 18

K. Two kinds of Supervised Contrastive Losses

• The authors use Lsup_out as their baseline because Lsup_in's normalization cannot correct positive-sample imbalance, and Lsup_in is more bias-vulnerable with inferior classification performance.Source: K. Two kinds of Supervised Contrastive… · p. 19

• The study is confined to image classification; extending FSCL to other tasks may require task-appropriate fairness notions and specialized architectures.Source: L. Discussion on Limitations · p. 19
• FSCL requires sensitive-attribute labels and cannot be used when those labels are entirely absent, though a few labels can reduce annotation costs.Source: L. Discussion on Limitations · p. 19