Source-linked AI summary
Towards Controllable Biases in Language Generation
Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, Nanyun Peng
TL;DR
Societal bias in NLG is difficult to analyze and mitigate, especially when generated text contains demographic inequalities. The paper adapts adversarial trigger search to induce specified bias polarities, compare demographic effects, and reduce disparities. The resulting framework supports bias analysis and mitigation across language and dialogue generation, while its controllability is limited by discrete vocabulary tokens.
Problem
Biases in NLG have been comparatively underexamined despite the social impact of applications involving direct human interaction.
Method
The framework adapts adversarial trigger search to prepend bias-control triggers that associate or dissociate demographic mentions with negative, neutral, or positive regard.
Results
The framework analyzes demographic inequalities and mitigates bias by equalizing bias polarity ratios while generating less negatively biased text across language and dialogue generation.
Takeaways & Limitations
Bias triggers provide a general framework for examining demographic inequalities and mitigating societal biases in NLG systems.
Takeaways & Limitations
Bias polarity ratios vary greatly across demographic pairs, partly because discrete vocabulary tokens make smooth bias control difficult.
Abstract
from arXiv · showhide
We present a general approach towards controllable societal biases in natural language generation (NLG). Building upon the idea of adversarial triggers, we develop a method to induce societal biases in generated text when input prompts contain mentions of specific demographic groups. We then analyze two scenarios: 1) inducing negative biases for one demographic and positive biases for another demographic, and 2) equalizing biases between demographics. The former scenario enables us to detect the types of biases present in the model. Specifically, we show the effectiveness of our approach at facilitating bias analysis by finding topics that correspond to demographic inequalities in generated text and comparing the relative effectiveness of inducing biases for different demographics. The second scenario is useful for mitigating biases in downstream applications such as dialogue generation. In our experiments, the mitigation technique proves to be effective at equalizing the amount of biases across demographics while simultaneously generating less negatively biased text overall.
1 Introduction
The paper introduces a general framework for controlling societal bias polarity in generated text conditioned on demographic mentions, supporting both bias analysis and mitigation. It uses adversarial triggers to compare demographic inequalities and reduce negative bias while equalizing bias levels.
- NLG bias research has received less attention than work on word embeddings and natural language understanding, despite direct human interaction in applications such as chatbots and translation.
- The framework induces negative, neutral, or positive regard toward specified demographics by optimizing bias-control triggers prepended to model inputs.Inputs contain demographic mentions and bias contexts that may influence generated outputs.
- Bias analysis identifies topics associated with demographic inequality and compares how difficult it is to induce negative bias across demographic groups.The paper reports more international-relations text for RACE-BLACK than RACE-WHITE under one trigger, and greater difficulty inducing negative bias toward RACE-WHITE than RACE-BLACK.
- Bias mitigation triggers equalize bias polarity ratios across demographics while generating less negatively biased text.
- Evaluations on GPT-2 and DialoGPT show that the framework can analyze and mitigate biases in language and dialogue generation.The experiments combine automatic and human, quantitative and qualitative evaluations, including a downstream dialogue-generation setting.
2 Problem Definition and Background
The paper defines societal bias in NLG through unequal social perceptions of demographic groups and measures it using regard. It operationalizes demographic groups as sets of referring surface forms and controls generated text with prepended triggers and bias contexts.
- A demographic group is a socially defined group represented in text by an equivalence class of mentions, such as man, male, and guy for GENDER-MALE.
- The paper defines societal bias as unequal social perceptions of demographic groups in the distributions of large sets of generated texts.
- Regard measures the general social perception toward a demographic group and can differ from sentiment scores.
- Adversarial triggers are token sequences prepended to input prompts to influence generated text, while bias contexts provide textual settings that may induce demographic bias.Prompts contain a demographic mention and a bias context, such as a template asking how a person was described or regarded.
3 Controllable Biases for NLG
The framework searches for triggers that associate or dissociate demographic groups with target regard polarities, enabling controlled bias generation and mitigation. It uses corpus-level language-model probabilities to optimize these objectives, including a mitigation formulation targeting less negative text for both demographics.
- Trigger Search Algorithm: Trigger search iteratively replaces tokens in a default trigger string to optimize an objective over the annotated corpus.The process terminates when no token replacement further improves the objective.
- Trigger Search Algorithm: The framework represents demographic mentions, target samples, regard polarities, triggers, and a trained language model within an annotated-corpus formulation.Each input prompt contains a demographic mention and bias context, while target samples are grouped by negative, neutral, or positive regard.
- Bias Association and Dissociation: Bias association maximizes the model probability of target samples for specified demographic-regard pairs, whereas dissociation minimizes that probability.The objective uses Fθ(Yr; ˜t, Xd), summed over corpus pairs containing demographic d and target samples with regard r.
- Bias Control Objectives: Linear combinations of demographic-regard association and dissociation terms let the trigger-search objective control multiple targeted specifications.The formulation can associate different demographics with different target regard samples.
- Bias Control Objectives: The bias-induction objective associates negative regard with one demographic and positive regard with another while dissociating the opposite pairings.This formulation supports diagnosing and analyzing biases by observing how strongly triggers influence generated text.
- Bias Control Objectives: The mitigation objective associates neutral and positive samples with both demographics and dissociates negative samples, empirically making bias amounts more equal.The formulation targets less negative text for both demographics, although it does not directly target their relative bias amount.
4 Evaluation of Bias Triggers
Automatic and human evaluations show that bias triggers can induce biases in specified directions and mitigate them while preserving comparable grammaticality. The evaluations also connect trigger effects to topical patterns and show that automatic and human regard assessments are aligned.
- Automatic Evaluation: Bias triggers successfully amplify biases in the original direction and induce biases in the opposite direction across gender, sexual orientation, and racial biases.The four conditions compare no trigger, mitigation, original-direction bias, and opposite-direction bias triggers using negative, neutral, and positive regard ratios.
- Automatic Evaluation: Mitigation triggers produce smaller negative regard gaps between demographic pairs than generated text without triggers.This supports the mitigation objective for downstream language-generation applications.
- Human Evaluation: Spearman’s correlation between automatic and human average regard scores is 0.69.Human annotations of 59–100 samples per condition show trends similar to the automatic evaluation.
- Human Evaluation: Grammaticality remains comparable across trigger conditions, with mitigation-trigger samples rated slightly more grammatical than samples generated without triggers.The grammaticality scale assigns 5 to samples with no grammatical errors.
- Diagnosing Biases: Mitigation triggers produce positive words for both demographics, while original-direction and opposite-direction triggers concentrate more negative words on Black and White, respectively.The topical analyses associate mitigation outputs with photography and nature, original-direction outputs with international relations and more negative words for Black, and opposite-direction outputs with games and slightly more negative words for White.
5 Bias Triggers for Dialogue Generation
The dialogue-generation experiments adapt bias triggers to demographic names and show that mitigation generalizes across names while improving regard scores and reducing disparities.
- The experiments use the pre-trained medium version of DialoGPT and replace general demographic strings with names for conversational evaluation.Sixteen randomly chosen names from each targeted demographic are used when searching mitigation triggers.
- z-score 22.7 and p-value 1.7 × 10−114 show that the observed result is statistically significant relative to random partitions.
- 16 names are cycled during mitigation-trigger search so the resulting triggers can generalize across different names.
- 0.53 average regard for Black names and 0.52 for White names are obtained when mitigation triggers are appended to names and bias contexts.
- The mitigation effects generalize to both names used to find the trigger and unseen names, with decreased score gaps and increased overall scores.
6 Related Work
Prior NLP bias research has focused mainly on embeddings and language understanding, while this work addresses the comparatively limited study of bias in generation using controllable objectives and triggers.
- Earlier social-bias research concentrated largely on word embeddings and natural language understanding rather than natural language generation.
- Existing work includes bias quantification, bias metrics and datasets, language-model regularization, and encoder-decoder systems for neutralizing subjective biases.
- Related control methods modify model behavior through training, weighted decoding, attribute classifiers, or other pipeline components.
- Although this paper uses gradient-based adversarial triggers, its objectives differ from prior triggers designed to generate racist output from arbitrary inputs.
7 Conclusion
The framework supports both analyzing demographic inequalities in generated text and mitigating biases across language and dialogue generation. The authors identify broader textual-context generalization as future work.
- The framework reveals learned model biases by examining topics associated with demographic inequality and comparing trigger effectiveness across demographics.
- Bias triggers can mitigate biases in both language generation and dialogue generation.
- Future work will investigate whether the framework generalizes to more variations in textual contexts.
A.1 Bias Triggers and Generated Text
The appendix tables distinguish triggers that create opposing demographic regard associations from mitigation triggers that move both demographics toward positive and neutral text.
- Table 4 lists triggers for conditions that associate one demographic with negative regard and the other with positive regard.The plus and minus signs indicate positive and negative objectives, respectively.
- Table 5 lists mitigation triggers that associate both demographics with positive and neutral regard samples while dissociating them from negative samples.
Trigger search algorithm.
The trigger-search experiments specify implementation settings, bias-context templates, and evaluation components for generating and assessing controlled biases. The setup combines sampled decoding, a regard classifier, and a dataset of labeled regard examples.
- Trigger search takes 1-2 hours on an RTX 2080Ti GPU, while α and β are manually tested over {0, 1, 2}.
- The regard dataset contains 117 negative, 93 neutral, and 92 positive target samples used in trigger search.
- Bias contexts replace “XYZ” with demographic mentions and are restricted to respect contexts for DialoGPT evaluation.
- The experiments use top-k sampling with k = 40, temperature = 0.7, and a maximum sequence length of 40.
- A three-instance regard-classifier ensemble assigns each sample its majority-predicted label.The classifier instances are separately fine-tuned on the same regard dataset.
A.3 More Language Generation Results
The additional language-generation results examine gender disparities, mitigation performance, human validation, annotation criteria, and trigger quality. They report a statistically significant gender gap, equalized mitigated regard, and stronger cross-demographic triggers when search uses diverse names.
- Gender bias results: The average regard score is 0.31 for female names and 0.35 for male names, producing a gender gap of 0.04.The gap is statistically significant, with z-score 12.2 and p-value 1.8 × 10−34.
- Gender bias results: The mitigation trigger yields an average regard score of 0.57 for both female- and male-name texts.The results generalize to unseen names, and the mitigation trigger produces higher average regard with generally smaller absolute differences.
- Human validation: Human annotation agreement reaches Cohen’s kappa 0.51 across 600 samples and 0.61 for samples labeled negative, neutral, or positive.Spearman’s correlation is 0.71 for the 461 classified samples and 0.66 between majority human labels and automatic labels.
- Human validation: The regard annotation task asks how a sentence affects most people’s social perception of [PERSON], including positive, negative, neutral, and mixed categories.
- Trigger quality: Triggers found using names from multiple demographic groups are more effective across groups than triggers searched with one group’s names.The authors also observe that punctuation-containing triggers can produce poorer-quality text, despite triggers sometimes remaining effective and grammatical.