PaperlayerView exact revision on arXiv

Source-linked AI summary

Contrastive Learning for Fair Representations

Aili Shen, Xudong Han, Trevor Cohn, Timothy Baldwin, Lea Frermann

arXiv:2109.10645v1cs.CLcs.AI

TL;DR

Classification models can learn biased representations and predictions from societal stereotypes in training data. The paper combines cross-entropy with contrastive losses based on task labels and protected attributes. Across four NLP and vision tasks, it reports the best accuracy–fairness tradeoff in each case.

  • Problem

    Training data can encode societal stereotypes, leading classifiers to learn spurious correlations with protected attributes and make unequal subgroup predictions.

    Source: Introduction · p. 1

  • Method

    The method combines cross-entropy with contrastive losses that organize representations by task labels and discourage reliance on protected attributes.

    Source: Introduction · p. 1

  • Results

    The method achieves the best accuracy–fairness tradeoff across four NLP and computer vision tasks.

    Source: Introduction · p. 1

  • Takeaways & Limitations

    The conclusion reports that both contrastive loss components are needed for bias reduction while maintaining main-task accuracy.

    Source: Conclusion · p. 7

    The method is designed to remove protected-attribute information assuming attackers use linear classifiers; protection against nonlinear classifiers remains future work.

    Source: Limitations · p. 7

Abstract

from arXiv · show

Trained classification models can unintentionally lead to biased representations and predictions, which can reinforce societal preconceptions and stereotypes. Existing debiasing methods for classification models, such as adversarial training, are often expensive to train and difficult to optimise. In this paper, we propose a method for mitigating bias in classifier training by incorporating contrastive learning, in which instances sharing the same class label are encouraged to have similar representations, while instances sharing a protected attribute are forced further apart. In such a way our method learns representations which capture the task label in focused regions, while ensuring the protected attribute has diverse spread, and thus has limited impact on prediction and thereby results in fairer models. Extensive experimental results across four tasks in NLP and computer vision show (a) that our proposed method can achieve fairer representations and realises bias reductions compared with competitive baselines; and (b) that it can do so without sacrificing main task performance; (c) that it sets a new state-of-the-art performance in one task despite reducing the bias. Finally, our method is conceptually simple and agnostic to network architectures, and incurs minimal additional compute cost.

Introduction

Trained classifiers can encode societal stereotypes through spurious correlations with protected attributes, while common debiasing approaches can be costly or difficult to optimise. The paper introduces a contrastive-learning method and reports a strong accuracy–fairness tradeoff across four NLP and vision tasks.

Introduction

  • Spurious correlations between target labels and protected attributes can produce unequal subgroup predictions and leak authorship-related sensitive information.Source: Introduction · p. 1
  • Data manipulation can be costly and ineffective, while adversarial training can be difficult to optimise and computationally expensive.Source: Introduction · p. 1
  • The method combines cross-entropy with two contrastive losses to learn representations that group by task label rather than protected attribute.Source: Introduction · p. 1
  • The method achieves the best accuracy–fairness tradeoff across four NLP and computer vision tasks.Source: Introduction · p. 1
  • The method is simple to implement, architecture-agnostic, and incurs minimal additional computing cost.Source: Introduction · p. 1

Related Work

The related work reviews debiasing methods and contrastive learning. It positions the proposed approach as an augmented training objective that adapts supervised contrastive learning to fair classification.

Related Work

  • The related-work section surveys debiasing methods and contrastive learning as the two areas most closely connected to the paper.Source: Related Work · p. 2

Debiasing Methods

  • Prior debiasing methods include input balancing, post-training removal of sensitive information, and augmented objectives such as adversarial training.The proposed method also augments the objective but adds no model parameters, making it computationally lighter than adversarial methods.Source: Debiasing Methods · p. 2

Contrastive Learning

  • Contrastive learning pulls similar instances together and pushes dissimilar ones apart; supervised contrastive learning defines pairs by class labels.The paper adapts supervised contrastive learning to fair supervised learning.Source: Contrastive Learning · p. 2
  • Figure 1 illustrates sentiment classification with cross-entropy loss and contrastive losses based on task labels and protected attributes.Source: Figure 1 · p. 2

Fair & Supervised Contrastive Learning

The method combines cross-entropy with two contrastive losses to separate representations by main-task class while discouraging differentiation by protected attributes. Its embedding and encoder architecture is compatible with language and vision tasks.

Fair & Supervised Contrastive Learning

  • The proposed method combines cross-entropy with supervised and fair contrastive losses to produce task-indicative, protected-attribute-agnostic representations.It is described as compatible with different classifier architectures and data modalities, including language and vision.Source: Fair & Supervised Contrastive Learning · p. 2

Cross-entropy Loss

  • Cross-entropy maximizes the predicted probability of the gold class but does not model similarity between instances in representation space.The method adds supervised contrastive learning to explicitly model that similarity.Source: Cross-entropy Loss · p. 2

Contrastive Losses

  • Supervised contrastive loss uses same-class instances as positives and other-class instances as negatives, encouraging distinct class clusters.Fair contrastive loss instead uses protected-attribute labels to construct pairs and aims to mix representations with different protected-attribute values.Source: Contrastive Losses · p. 3

Objective Function

  • The objective weights cross-entropy and the difference between supervised and fair contrastive losses, with α and β controlling their relative importance.The supervised term pulls same-task-label instances together and pushes different-class instances apart; the fair term encourages same-protected-attribute instances to stay apart and different-class instances to mix.Source: Objective Function · p. 3
  • The end-to-end and pipelined versions differ in whether the encoder and classifier are trained together or in stages.The method is presented as avoiding data manipulation, post-processing, and an additional adversarial network, and as more efficient to train than adversarial debiasing.Source: Objective Function · p. 3

Experiments

Experiments cover four datasets spanning NLP and vision, binary and multiclass classification, with accuracy, fairness, and efficiency measures. Models are compared against unmitigated, post-processing, and adversarial baselines.

Experiments

  • Models use varied embedding architectures and a two-layer encoder; hyperparameters are tuned on development data, with selection balancing GAP and near-best accuracy.The authors note the absence of a standardized model-selection method for fairness research.Source: Experiments · p. 3

Baselines

  • The baselines include cross-entropy without mitigation, INLP projection, and adversarial training with three protected-attribute discriminators.For hate speech detection and activity recognition, embeddings are fixed after pretraining or fine-tuning.Source: Baselines · pp. 3–4

Evaluation Metrics

  • Accuracy evaluates the main task, while GAP measures subgroup differences in true-positive rates; the reported GAP formulation assumes binary protected attributes.For multiclass main-task labels, the classwise GAP values are combined using a root mean square.Source: Evaluation Metrics · p. 4
  • Linear leakage measures how well a linear classifier recovers protected attributes from hidden representations or logits, with lower values indicating fairer models.Tradeoff aggregates normalized accuracy, GAP, and leakage measures with equal weight for predictive performance and overall fairness; efficiency is average GPU training time over 10 runs.Source: Evaluation Metrics · p. 4

Experiments

  • Experiments span four datasets across NLP and vision, including binary and multiclass classification, and report accuracy, fairness, and efficiency.Detailed accuracy–fairness tradeoffs are explored for Moji and imSitu.Source: Evaluation Metrics · p. 4

Experiment 1: Twitter Sentiment Analysis

On Moji, the proposed contrastive models deliver the largest accuracy gains and bias reductions across metrics, with leakage near the chance-level lower bound. The authors hypothesize that debiasing improves robustness to training-set skew relative to the balanced test set.

Task and Dataset

  • The task predicts binary sentiment in English tweets; the private binary attribute indirectly captures authors’ race as AAE or SAE.Training data are balanced separately for sentiment and race but skewed in their combinations.Source: Task and Dataset · p. 4

Implementation Details

  • DeepMoji, pretrained on 1.2 billion English tweets, supplies text representations, and its parameters remain fixed during the experiments.The model is used as Embed(·).Source: Implementation Details · p. 4

Results

  • Con∗ and Conft∗ deliver the largest accuracy gains and bias reductions across metrics, with leakage around 55 approaching the chance-level value of 50 and the best accuracy–fairness Tradeoff.The reported results are in Table 1 (Moji).Source: Results · p. 4, Table 1 · p. 5
  • The authors hypothesize that debiasing acts as regularization, improving robustness to training-set sentiment–race skew relative to the balanced test set.The training set is skewed across sentiment–race combinations, whereas dev and test sets are balanced.Source: Results · pp. 4–5, Task and Dataset · p. 4
  • On Moji, Con∗ achieves the best Accuracy and Leakage@h, while INLP and Adv attain lower Leakage@h at an Accuracy cost as hyperparameters vary.Red-circled points mark each model’s Pareto frontier.Source: Results · p. 5

Experiment 2: Hate Speech Detection

The experiment evaluates hate-speech classification while debiasing for age, the protected attribute reported to have the highest model bias in prior work. The contrastive variants perform best on Leakage@h and overall Tradeoff, with INLP leading other bias metrics.

Task and Dataset

  • The task predicts whether an English tweet contains hate speech, with age selected for debiasing among binary gender, race, and age attributes.Prior work identified age as producing the highest model bias; the standard split has 31K/7K/7K train/dev/test instances.Source: Experiment 2: Hate Speech Detection · p. 5

Implementation Details

Results

  • Con∗ and Conft∗ match INLP on GAP, lead all methods on Leakage@h, and achieve the best overall Tradeoff; INLP leads other bias metrics, while Adv reduces bias only slightly.The reported results are on the hate-speech test set.Source: Experiment 2: Hate Speech Detection · p. 6

Experiment 3: Profession Classification

In profession classification across 28 classes, Con∗ achieves the best overall Tradeoff, while the results show that bias reduction can come with a substantial Accuracy cost. Adv and Con∗ also marginally increase GAP.

Task and Dataset

Implementation Details

Results

  • Con∗ achieves the best overall Tradeoff, while Conft∗ substantially reduces bias across three fairness metrics at the cost of a large Accuracy drop.INLP has the best GAP result; Con∗ leads Leakage@ŷ at accuracy similar to CE, and Adv leads Leakage@h.Source: Experiment 3: Profession Classification · p. 6
  • Adv and Con∗ marginally increase GAP; the authors hypothesize that 28 task classes inhibit joint adversarial and contrastive training from mitigating bias.The proposed explanation is specific to this multi-class setting.Source: Experiment 3: Profession Classification · p. 6

Experiment 4: Activity Recognition

For imSitu activity recognition, Con∗ achieves the best overall Tradeoff, while Conft∗ leads on GAP and Leakage@ŷ. The task groups fine-grained activity labels into 12 classes to make the dataset more practical for training and debiasing.

Task and Dataset

  • The experiment predicts image activities from imSitu; its 211 fine-grained activity classes are grouped into 12 coarse-grained labels, yielding 12K/3K/2K train/dev/test instances.The original fine-grained dataset has roughly 110 instances per activity, making strong performance and debiasing difficult without grouping.Source: Experiment 4: Activity Recognition · p. 6

Implementation Details

  • A pretrained ResNet-50 is finetuned, and its best development-set snapshot supplies 2,048-dimensional image representations.The classifier layer is first trained before finetuning the backbone.Source: Experiment 4: Activity Recognition · p. 6

Results

  • Con∗ achieves the best overall Tradeoff; Conft∗ leads on GAP and Leakage@ŷ, which the authors attribute to disconnecting the main-task classifier from encoder training.Con∗ also leads on Leakage@h, while INLP leads on Leakage@ŷ among the compared baselines.Source: Experiment 4: Activity Recognition · p. 6
  • At higher Accuracy, neither Adv nor Con∗ reduces bias substantially over INLP in the Accuracy–Leakage@h comparison.Figure 2 plots the tradeoff on the test set.Source: Experiment 4: Activity Recognition · p. 6

Analysis

The analysis finds that the full contrastive model performs strongly across the reported metrics and that its representations cluster by sentiment while mixing race groups. Bias reduction is less impressive in multi-class settings, while a nurse-versus-surgeon binary setting shows stronger reductions than the multi-class setting.

Analysis

  • The ablations on Moji and Bios remove one contrastive loss component at a time to explore their individual impacts.The variants are denoted Conce+scl and Conce−fcl.Source: Effect of Loss Components · p. 6

Visualising Representations

  • In the t-SNE plots, Con∗ representations cluster by sentiment, while instances from different race groups mix.The plots compare CE and Con∗ on 100 random Moji samples from each main-task class; colors encode sentiment and race groups.Source: Visualising Representations · pp. 6–7, Figure 4 · p. 7

Effect of Loss Components

  • Con∗ achieves the best performance across all evaluation metrics on Moji, and the best accuracy while roughly equalling the best bias results on imSitu.The authors attribute this advantage to incorporating both contrastive loss components.Source: Effect of Loss Components · p. 6, Figure 3 · p. 7

Binary Classification vs. Multi-class Classification

  • Con∗ reduces bias on binary Moji and Hate speech tasks, while results are less impressive on multi-class Bios and imSitu, where separating more classes is harder.The authors link scattered main-task representations in multi-class settings to less effective bias reduction.Source: Binary Classification vs. Multi-class… · p. 7
  • For nurse-versus-surgeon Bios classification, switching to binary classification raises accuracy and slightly lowers GAP and Leakage@ŷ; Con∗ reduces gender bias more effectively than INLP or Adv, with a small accuracy decrease.The comparison uses the same nurse and surgeon data in multi-class and binary settings.Source: Binary Classification vs. Multi-class… · p. 7, Table 2 · p. 7

Limitations

The method targets protected-attribute information under a linear-attacker assumption. The authors identify non-linear attacks, end-to-end embedding fine-tuning, and informative multi-class negative samples as future-work directions.

Limitations

  • The approach is designed to remove protected-attribute information assuming the attacker is a linear classifier.Protection against non-linear classifiers is left for future work.Source: Limitations · p. 7
  • The embedding module is not learned or fine-tuned end-to-end with the encoder and classification layer.The authors suggest that fine-tuning embeddings could yield more task-specific or semantic-preserving representations and further remove pretrained-model biases.Source: Limitations · p. 7
  • Constructing informative negative samples for multi-class classification remains an open direction identified by the authors.Source: Limitations · p. 7

Conclusion

The conclusion presents contrastive losses as a way to reduce bias while maintaining task accuracy. Experiments across NLP and vision datasets support the method's effectiveness, and the analyses emphasize the role of both contrastive components.

Conclusion

  • The method combines cross-entropy with two contrastive losses, pulling together instances from the same main-task class and pushing apart instances sharing a protected attribute.Source: Conclusion · p. 7
  • Experiments across four NLP and vision datasets demonstrate the proposed method's effectiveness.Source: Conclusion · p. 7
  • Further analysis and ablations indicate that both contrastive components are needed for bias reduction while maintaining main-task accuracy.Source: Conclusion · p. 7
Loading 2109.10645v1…