Reducing Sentiment Bias in Language Models via Counterfactual Evaluation

Po-Sen Huang; Huan Zhang; Ray Jiang; Robert Stanforth; Johannes Welbl; Jack Rae; Vishal Maini; Dani Yogatama; Pushmeet Kohli

Source-linked AI summary

Reducing Sentiment Bias in Language Models via Counterfactual Evaluation

Po-Sen Huang, Huan Zhang, Ray Jiang, Robert Stanforth, Johannes Welbl, Jack Rae, Vishal Maini, Dani Yogatama, Pushmeet Kohli

arXiv:1911.03064v3Nov 8, 2019cs.CLcs.CYcs.LG

TL;DR

Language models may produce different sentiment when sensitive attributes in a conditioning context change, raising a fairness question for generated text. The paper evaluates this counterfactually with fairness metrics and latent-representation regularization, finding reduced sentiment bias while maintaining comparable perplexity and semantic similarity.

Problem
The paper asks whether sensitive attributes in conditioning contexts systematically affect generated-text sentiment, a fairness concern as language models internalize social biases.
Method
It replaces sensitive tokens to form counterfactual inputs, compares sentiment distributions with individual and group Wasserstein-based fairness metrics, and regularizes latent representations.
Results
The study quantifies counterfactual sentiment bias and finds that embedding and sentiment prediction-derived regularization reduce it while retaining comparable perplexity and semantic similarity.
Takeaways & Limitations
The framework supports evaluating and reducing generated-text sentiment bias under specified sensitive attributes and can extend to other specifications and measures.
Takeaways & Limitations
Template-based evaluation can lack context sensitivity and is limited rather than comprehensive.

Abstract

from arXiv · show

Advances in language modeling architectures and the availability of large text corpora have driven progress in automatic text generation. While this results in models capable of generating coherent texts, it also prompts models to internalize social biases present in the training corpus. This paper aims to quantify and reduce a particular type of bias exhibited by language models: bias in the sentiment of generated text. Given a conditioning context (e.g., a writing prompt) and a language model, we analyze if (and how) the sentiment of the generated text is affected by changes in values of sensitive attributes (e.g., country names, occupations, genders) in the conditioning context using a form of counterfactual evaluation. We quantify sentiment bias by adopting individual and group fairness metrics from the fair machine learning literature, and demonstrate that large-scale models trained on two different corpora (news articles, and Wikipedia) exhibit considerable levels of bias. We then propose embedding and sentiment prediction-derived regularization on the language model's latent representations. The regularizations improve fairness metrics while retaining comparable levels of perplexity and semantic similarity.

1 Introduction

Large language models can internalize social biases, including systematic sentiment differences caused by sensitive attributes in conditioning contexts. The paper proposes metrics and regularization methods to quantify and reduce this counterfactual sentiment bias.

Motivation: Generated sentiment can differ systematically when an occupation in the conditioning context changes from “baker” to “accountant”.GPT-2 generated more positive sentiment for “baker” and more negative sentiment for “accountant”.
Motivation: Counterfactual evaluation varies sensitive attributes such as countries, occupations, and names while holding other context constant.The resulting sentiment differences suggest sentiment bias relevant to fairness in downstream applications.
Contributions: Embedding and sentiment prediction-derived regularization reduce sentiment bias while retaining comparable perplexity and semantic similarity.The experiments also identify a trade-off between fairness and semantic relevance.
Contributions: The paper proposes individual and group fairness metrics to quantify counterfactual sentiment bias in generated text.These metrics are drawn from the fair machine learning literature.
Contributions: The framework is presented as generalizable beyond sentiment and the studied country, occupation, and name attributes.The authors describe it as a general framework for reducing bias under a specification measure.
Contributions: Automatic metrics and human evaluations of sentiment and semantic relevance show a strong correlation.Both evaluation modes are used to assess the proposed methods.

2 Background & Related Work

Prior work documents social and demographic biases in NLP systems and develops mitigation methods, especially for language-model probabilities and representations. This paper instead probes generated text with sentiment analysis and targets counterfactual bias under specified fairness measures.

Bias in NLP systems: NLP systems have exhibited cultural associations and demographic imbalances across tasks including gendered coreference, semantic role labeling, and image captioning.The related work situates language-model bias within broader systematic imbalances in NLP.
Bias mitigation: Counterfactual data augmentation has been used to mitigate occupation-specific gender bias while better retaining language-model performance than embedding debiasing.The cited work compares augmentation with debiasing word embeddings.
Bias mitigation: Prior studies found gender bias in pretrained language-model representations, but did not consider language generated from those models.The paper distinguishes representation bias from bias in generated text.
This paper’s distinction: This paper probes generated texts with sentiment analysis rather than word-generation probabilities such as occupation ratios.It also proposes a framework covering specifications such as country names, occupations, and gendered names.
Experimental setting: The experiments investigate Transformer models with approximately 708 million parameters trained on English news articles and WikiText-103.These models are described as having a similar parameter count to GPT-2.
Fairness background: Fairness research includes individual and group criteria, while causal-inference tools extend fairness analysis through causal graphs.The paper’s counterfactual framing relates to this broader fairness literature.

3 Counterfactual Evaluation of Sentiment Bias

The paper evaluates sentiment bias by replacing sensitive tokens in matched inputs and comparing the resulting sentiment-score distributions. It uses Wasserstein-1 distance to define individual and group fairness metrics without requiring a fixed sentiment threshold.

Counterfactual construction: A counterfactual input replaces all sensitive tokens φ(a) with corresponding tokens φ(ã), leaving non-sensitive words unchanged.The specification defines sensitive-attribute values and their token sets, such as gendered terms.
Counterfactual construction: Generated sentiment is modeled as S(x)=f_s(LM(x)), with distribution P_S(x) over scores in [0,1].LM samples a sentence conditioned on x, and the sentiment classifier supplies its score.
Wasserstein fairness: Wasserstein-1 distance compares original and counterfactual sentiment distributions, averaging threshold disparities over uniformly random τ∈[0,1].This avoids making fairness dependent on a predetermined decision threshold.
Wasserstein fairness: The Figure 1 sentiment distributions have Wasserstein-1 distance 0.13.Figure 2 uses truncated Gaussian distributions on [0,1] to illustrate the metric.
Fairness metrics: Average Individual Fairness averages Wasserstein-1 distances between each evaluation sentence and its counterfactual across M templates.The evaluation uses unordered pairs of distinct sensitive-attribute values.
Fairness metrics: Average Group Fairness averages Wasserstein-1 distances between each sensitive subgroup’s sentiment distribution and the entire evaluation-set distribution.Sentences are assigned to disjoint subgroups based on sensitive tokens.

4 Language Models with Fair Sentiment Distribution

The paper introduces two hidden-representation regularization methods to reduce counterfactual sentiment bias: embedding regularization and sentiment regularization. Both are evaluated within a three-step curriculum that adds fairness training while retaining ordinary language-model training.

Counterfactual Evaluation: The framework evaluates counterfactual prefixes by replacing a sensitive token with one from another subgroup.The original and perturbed prefixes differ only in the sensitive token, enabling subgroup comparisons.
Counterfactual Evaluation: The target is similar sentiment distributions for original and perturbed prefixes, approximated through hidden features because sequence sampling is prohibitively expensive.The desired distributional similarity is expressed with Wasserstein-1 distance, while hidden states proxy future-generation distributions.
Embedding Regularization: Embedding regularization minimizes cosine distance between the original and perturbed hidden embeddings, whose similarity influences their future-token distributions.The method averages the last two embedding vectors to capture higher-level semantics such as sentiment.
Limitations: Embedding regularization can be too strong because it lacks specification-measure information, potentially causing the model to ignore sensitive tokens and harming contextual conditioning.The paper notes that this extreme fairness case may reduce model performance.
Sentiment Regularization: Sentiment regularization applies a sentiment classifier to both hidden representations before minimizing the distance between the classifier outputs.This approach targets sentiment-related differences rather than directly forcing all hidden representations to match.
Curriculum Training: The three-step curriculum trains the language model, trains a sentiment classifier from extracted features, and then performs debiasing with an added fairness loss.The third step continues training on examples containing sensitive tokens while retaining the regular language-model objective.

5 Experiments

The experiments evaluate counterfactual sentiment bias across two corpora, sensitive attributes, classifiers, and regularization strengths, while measuring fairness and generation quality. Both regularization methods generally improve fairness, with sentiment regularization offering stronger fairness–semantic similarity trade-offs.

Experimental setup: Two TransformerXL language models are trained on WikiText-103 and WMT-19, with embedding- and sentiment-regularization losses evaluated at multiple λ values.The models are similar in scale to GPT-2, and validation losses determine the reported regularization strengths.
Experimental setup: Counterfactual evaluation replaces sensitive tokens in 10 templates, samples 1,000 continuations per prefix, and scores them with external sentiment classifiers.The study considers Country, Occupation, and Name attributes, using three sentiment classifiers and fairness metrics based on their scores.
Evaluation scope: Template-based evaluation is necessarily limited because fixed templates can lack context sensitivity and do not provide comprehensive coverage.The templates are evaluation-only rather than training data, so the models must generalize to them.
Fairness results: Both proposed approaches reduce individual and group fairness scores relative to baseline models, and larger λ typically produces further fairness improvements.The overall pattern is reported for Occupation and is similar for the Country and Name attributes.
Fairness–quality trade-off: The experiments observe a trade-off between fairness and semantic similarity, alongside evaluation of perplexity and semantic relevance.Perplexity is measured on full test sets and sensitive-token subsets, while semantic similarity uses cosine similarity with a 0.4 threshold.
Fairness–quality trade-off: Sentiment regularization achieves larger individual-fairness improvements than embedding regularization while maintaining similar semantic similarity.Both methods improve individual fairness compared with baseline models, but the sentiment-based method is more favorable at comparable relevance levels.

6 Conclusion

The paper quantifies counterfactual sentiment bias in large-scale language-model text and demonstrates methods that reduce it while preserving generation quality. It also identifies extensions to other sensitive attributes and specifications.

The study quantifies sentiment bias in texts generated by large-scale language models.
Two flexible methods reduce counterfactual sentiment bias while maintaining similar perplexity and generation semantics.
The framework could be extended to other sensitive attributes, cross-subgroups, and specification measures beyond sentiment.

A Sensitive Attributes and Templates in Fairness Specification

The fairness specification defines sensitive attribute values and constructs evaluation templates for country, occupation, and name attributes. Country values are selected using baseline-generated sentiment patterns, while occupation and name lists follow specified source or sampling procedures.

The study considers Country, Name, and Occupation as sensitive attributes for sentiment-bias evaluation.
Country: Country tokens are selected after evaluating the baseline model for contexts with especially positive or negative sentiment.
Country: The Country attribute uses ten specified country names, filled into each evaluation template.
Occupation: Occupation tokens come from a list of occupation names introduced by Zhao et al. (2018) and are inserted into each template.
Name: Name tokens are randomly selected common male and female names from different US ethnicity groups and used in the name templates.

B Additional Experimental Details

Additional experiments describe the datasets, model architectures, curriculum-training stages, sentiment projection, sampling procedure, and quality evaluation. The debiasing stage adds fairness regularization while retaining scalability and comparable generation quality.

Dataset: WikiText-103 contains 28,591 Wikipedia articles and over 100 million tokens, while WMT-19 contains 14,635,198 English news articles.
Language model architectures: The WikiText-103 model has 257M parameters, and the WMT-19 model has 708 million parameters.
Training: Curriculum training includes language-model training, sentiment-projection training, and language-model debiasing.
Sentiment projection training: The sentiment classifier is a 3-layer MLP with hidden-layer size 128, trained on examples with absolute sentiment scores greater than 0.7.
Language model debiasing: The debiasing approach adds sentiment projection and regularization losses while remaining scalable to large language models.
Sample generation: Models generate 1,000 sentences per template, using ten templates per sensitive attribute, maximum length 50 tokens, and temperature 1.0.

C.1 Results on the Occupation attribute with the Google Cloud sentiment API

Additional occupation experiments evaluate individual and group fairness on WMT-19 and WikiText-103 using Google Cloud sentiment scores. The observed trends match the main results, with both proposed methods improving fairness metrics.

Figure 7 reports individual and group fairness scores for Occupation on WMT-19 and WikiText-103.
Both proposed methods effectively improve fairness metrics under the Google Cloud sentiment API evaluation.

C.2 Results on the Country attribute

Country-attribute experiments show that embedding and sentiment regularization reduce individual and group fairness scores across WMT-19 and WikiText-103 models. Larger regularization generally improves fairness, while perplexity remains nearly unchanged but excessive regularization can reduce semantic similarity.

Fairness improvements: Fairness improves as the regularization parameter increases for the country-attribute models.This trend is reported for both individual and group fairness evaluations across the sentiment classifiers.
Utility: Perplexity on the test set and sensitive-token subset remains almost unchanged after regularization.The evaluation tracks both overall test perplexity and perplexity on examples containing sensitive tokens.
Utility: Excessively large regularization can decrease semantic similarity between the sensitive token and generated text.Thus, fairness improvements may trade off against contextual relevance when the regularization parameter is too large.

C.3 Results on the Name attribute

Name-attribute experiments reproduce the fairness improvements observed for other attributes across WMT-19 and WikiText-103 models. The comparisons also include vanilla GPT-2, whose fairness scores are worse in most cases, while excessive regularization can produce irrelevant text.

Fairness results: Name-attribute models show effective fairness improvements under embedding and sentiment regularization across WMT-19 and WikiText-103 settings.The reported individual and group fairness results are consistent with those for the Country and Occupation attributes.
Model comparison: Vanilla GPT-2 exhibits larger, worse individual and group fairness scores than the comparison models in most attribute cases.The comparison covers Country, Occupation, and Name attributes against WikiText-103 and WMT-19 baselines.
Trade-offs: λ = 1000 produces nearly identical outputs for different occupations but makes the generated text irrelevant to the occupation context.This example illustrates an extreme fairness–semantic relevance trade-off, with S.S.=4.9 and S.S.c=1.1.
Evaluation: Semantic similarity is measured as the fraction of generated continuations whose universal-sentence-encoder cosine similarity exceeds 0.4.The threshold determines whether a continuation counts as semantically similar to its prefix.

D Human Evaluation Details

Human evaluation measures sentiment and semantic relevance of generated continuations, using multiple annotators and categorical judgments. The study compares these human judgments with automatic sentiment and semantic-similarity metrics.

Evaluation setup: Nineteen annotators labeled sentiment and semantic relevance, with each of 600 evaluated sentences judged by two annotators.Average Cohen’s kappa was 0.47 for sentiment and 0.45 for semantic relevance, indicating moderate inter-annotator agreement.
Sentiment evaluation: Sentiment annotations use four categories, while mixed positive-and-negative sentences are excluded from the automatic-correlation analysis.The retained labels are mapped to Negative, Neutral, and Positive scores of 0, 0.5, and 1.
Metric validation: Spearman’s correlation is reported between automatic scores and averaged human judgments for both sentiment and semantic relevance.The evaluation samples 100 generated sentences for these correlation analyses.