PaperlayerView exact revision on arXiv

Source-linked AI summary

Societal Biases in Language Generation: Progress and Challenges

Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, Nanyun Peng

arXiv:2105.04054v3cs.CL

TL;DR

NLG can produce societal biases that disproportionately affect marginalized populations, and direct user interaction makes these harms important to quantify. The paper surveys how data and techniques contribute to bias, adds experiments on decoding techniques, and identifies scope boundaries and open challenges. It highlights vulnerability impacts and shows that decoding-related bias findings must be considered alongside generated-text diversity.

  • Problem

    Societal biases in language generation can disproportionately harm marginalized populations, while decoding-related bias effects and broader NLG bias evidence remain insufficiently studied.

  • Method

    The paper provides a comprehensive survey of societal biases in NLG and experimentally evaluates multiple decoding techniques with existing bias metrics across GPT, GPT-2, and XLNet.

  • Results

    The study finds that decoding techniques differ in bias-related scores, with less diverse search techniques obtaining better scores for individual fairness, group fairness, and gendered word co-occurrence ratios.

  • Takeaways & Limitations

    Bias analysis in NLG should consider data, model and decoding components together, while treating text diversity as a potential confounding factor in bias metrics.

  • Takeaways & Limitations

    The survey’s scope excludes several other NLG tasks, including story, poetry, abstractive summarization, and paraphrase generation, because societal-bias research for them is not yet well established.

Abstract

from arXiv · show

Technology for language generation has advanced rapidly, spurred by advancements in pre-training large models on massive amounts of data and the need for intelligent agents to communicate in a natural manner. While techniques can effectively generate fluent text, they can also produce undesirable societal biases that can have a disproportionately negative impact on marginalized populations. Language generation presents unique challenges for biases in terms of direct user interaction and the structure of decoding techniques. To better understand these challenges, we present a survey on societal biases in language generation, focusing on how data and techniques contribute to biases and progress towards reducing biases. Motivated by a lack of studies on biases from decoding techniques, we also conduct experiments to quantify the effects of these techniques. By further discussing general trends and open challenges, we call to attention promising directions for research and the importance of fairness and inclusivity considerations for language generation applications.

1 Introduction

NLG has become more capable through advances in large pretrained models and expanding data, while its direct user interactions heighten the importance of understanding societal biases. The paper surveys these biases and examines how NLG techniques, including decoding, contribute to them.

  • 1 Introduction: Advances in Transformer-based models and expanding data repositories have enabled more capable NLG applications.Examples include virtual assistants, chatbots, translators, summarizers, and creative language composers.
  • 1 Introduction: Direct interaction with users means biased NLG can disproportionately harm marginalized populations and discourage their engagement with applications.The paper gives the example of an educational chatbot producing more negative responses about a specific ethnicity.
  • 1 Introduction: The paper presents a comprehensive survey of societal biases in language generation and reviews progress in bias analysis and mitigation.It focuses on NLG tasks that generate text sequences.
  • 1 Introduction: The paper also conducts an experimental study quantifying how different decoding techniques affect biases in generated text.Its contributions cover bias sources, decoding techniques, progress, challenges, and open problems.
  • 1 Introduction: The survey positions its focus on NLG tasks relative to prior surveys that largely address NLU or narrower bias topics.Its scope is limited to traditional autoregressive models.

2 Language Generation Tasks

The paper organizes NLG into continuation and transformation tasks, then reviews bias research across their major forms. Several other generation tasks remain insufficiently studied for societal biases.

  • 2 Language Generation Tasks: NLG tasks broadly divide into continuation generation conditioned on prompts and transformation generation that converts text into a targeted form.Table 1 organizes existing bias studies across these task categories.
  • 2.1 Continuation Generation Tasks: Continuation tasks include autocomplete and dialogue generation, both aiming to produce coherent, relevant text from conditioning inputs.Dialogue systems directly interact with users in domains such as health, customer service, and flight booking.
  • 2.2 Transformation Generation Tasks: Transformation tasks include machine translation and text rewriting, which preserve or modify text toward targeted properties.Translation preserves meaning across languages, while rewriting revises words or phrases for attributes such as neutrality, gender neutrality, or agency.
  • 2.2 Transformation Generation Tasks: Bias studies in machine translation have focused predominantly on gender associations arising across languages with different grammatical gender systems.Researchers have examined both commercial and academic translation systems.
  • 2.3 Other Tasks: Story, poetry, abstractive summarization, and paraphrase generation are not yet well-studied in the context of societal biases.The paper identifies this as a limitation of existing coverage.

3 Biases and their Negative Impacts

Bias in NLG is difficult to define and measure because generation is stochastic, open-ended, and lengthy, so studies often use demographic-inequality proxies. The paper surveys representational, allocational, and vulnerability impacts, with vulnerability linking generation to manipulation and harm.

  • 3.1 Bias Definitions and Metrics: NLG bias metrics are challenging because stochastic, open-ended, lengthy outputs do not directly fit traditional algorithmic fairness definitions.Existing work therefore commonly defines bias loosely as demographic inequality and uses intermediate proxy metrics.
  • 3.1 Bias Definitions and Metrics: Common proxies include regard ratios, sentiment ratios, fairness through sentiment scores, and gendered word co-occurrence measures.These metrics compare demographic associations in generated text or prompts.
  • 3.1 Bias Definitions and Metrics: Most existing metrics study biases toward people described in text, while fewer examine people who produce or receive the text.The latter associations are represented by comparatively fewer studies.
  • 3.2 Negative Impacts: Representational impacts include propagated stereotypes, misrepresentations, and denigrations, but their effects are difficult to quantify.The paper notes limited examples motivating existing NLG bias studies.
  • 3.2 Negative Impacts: Allocational impacts involve unequal resource allocation, while current NLG bias studies mostly analyze direct representational consequences.The paper notes that effective large-Transformer NLG techniques are relatively new and that norms are needed as adoption expands.
  • 3.2 Negative Impacts: Vulnerability impacts arise when open-domain generation amplifies a group’s susceptibility to manipulation and harm, including through privacy issues, misinformation, or radicalization.These vulnerabilities can increase susceptibility to representational and allocational impacts.

4 Contributors to NLG Biases

Biases in NLG can propagate through every stage of the pipeline, from data and model architecture to decoding, evaluation, and deployment. The paper surveys these contributors and experimentally shows that decoding choices affect measured bias, with less diverse methods often receiving better fairness scores.

  • Biases can propagate through data collection, model architecture, decoding, evaluation, and deployment in NLG pipelines.
  • 4.1 Biases from Data: Large web-trained language models learn from data containing gender, religion, and ethnic biases, while filtering offensive content can remove marginalized groups’ discourse.
  • 4.2 Biases from Model Architecture: Architectural studies report more gender bias in larger GPT-2 models, concentration in a small number of neurons and attention heads, and amplified bias after distillation.
  • 4.3 Biases from Decoding: Search and sampling methods produce similar relative-score outcomes, while beam search is most unbiased on absolute metrics, followed by greedy, top-k, and nucleus sampling.
  • A Study on Biases from Decoding: Less diverse decoding methods receive better individual-fairness, group-fairness, and gendered-word-co-occurrence scores, although existing metrics omit diversity.
  • 4.4 Biases from Evaluation: NLG evaluation can be biased through perplexity models trained on predominantly non-AAE data and through human annotator choices shaped by differing norms and linguistic backgrounds.
  • 4.5 Biases from Deploying Systems: Deployment can create feedback loops: internet access favors privileged languages, while unsuccessful users provide less feedback, reinforcing effectiveness disparities.
  • 4.5 Biases from Deploying Systems: Targeted data and feedback collection from excluded populations is proposed, but co-design must account for community goals and other factors to avoid additional harm.

5 Progress, Trends, and Challenges

The paper reviews data-, training-, and inference-time approaches for analyzing and mitigating bias in NLG, while identifying challenges in standardizing datasets and evaluating open-domain systems. Decoding-based methods are highlighted as a promising alternative because they can control generation without additional training.

  • 5.1 Data Methods: Data-based analysis and mitigation commonly use counterfactual prompts, balanced demographic datasets, and fine-tuning or training smaller models.
  • 5.1 Data Methods: Standardized datasets are difficult to create because large models and varying bias definitions must generalize across biases and demographics.
  • 5.1 Data Methods: Whether data-based mitigation transfers effectively from constrained settings to open-domain NLG remains unresolved.
  • 5.2 Training Methods: Training-based mitigation includes regularization and conditional control codes that guide generation toward specified bias-related targets.
  • 5.3 Inference Methods: Inference-time bias-mitigation research is sparse, but decoding-based methods can work with any pre-trained generation model without additional training.
  • 5.4 Evaluation: Evaluation distinguishes absolute scores, which summarize demographic inequalities, from relative scores, which explicitly compare all demographics.
  • 5.4 Evaluation: Transformation tasks tend to use absolute metrics, whereas less-constrained continuation tasks more often use flexible relative comparisons.

6 Open Problems and Proposals

The paper proposes open problems for reducing societal biases in language generation, spanning data curation, trade-offs in evaluation, interactive learning, and attention to negative impacts.

  • Bias-Aware Data Curation: Training datasets can encode bias through limited collection practices, including English-centric and Western-media sources, while bias datasets may represent only binary genders.The paper suggests diversifying datasets with viewpoints from more groups.
  • Understanding Trade-Offs: Existing analysis, mitigation, and evaluation methods are often small-scale and limited to a few biases and tasks.The paper calls for methods that generalize across diverse biases and contexts.
  • Understanding Trade-Offs: Bias metrics should jointly reduce bias and preserve desired text qualities such as diversity and fluency.
  • Interactive and Continuous Learning: Interactive and continuous learning could incorporate diverse opinions about fair and unfair generations over time.A unified framework could support more comprehensive evaluations for large-scale impact studies.
  • Focusing on Negative Impacts: Because few studies meaningfully engage with negative impacts, reframing evaluation around harms may produce metrics better aligned with reducing those impacts.Relative bias metrics may better reflect harms experienced by particularly impacted groups.

Ethics and Broader Implications

The paper surveys societal biases in language generation while acknowledging limitations in its data coverage, cultural scope, and potential misuse of decoding experiments.

  • Data: The survey does not assess the quality of datasets used to train popular language-generation models because of limited availability and size.It notes that large minimally filtered datasets can contain problems identified by other works.
  • Data: Many surveyed datasets and metrics approximate binary genders with gender-typical names, and most bias-evaluation data cover only binary genders.The paper suggests reformulating these resources to avoid harms and represent genders more accurately.
  • Techniques: Most surveyed techniques are trained or bias-tested on Western sources or culture, and bias techniques do not always transfer across cultures.
  • Experiments: The decoding experiments could potentially fuel misuse by adversarial actors.

A.1 Evaluating Biases Across Decoding Techniques and Metrics

The experiments compare decoding techniques across several language models and bias metrics, showing that decoding choices affect both measured bias and generated-text diversity.

  • Experimental Setup: The study evaluates autocomplete generations from GPT, GPT-2, and XLNet using beam search, top-k sampling, nucleus sampling, and related decoding methods.It uses b = 16 for beam search, k = 40 with temperature 0.7 for top-k sampling, and p = 0.95 for nucleus sampling.
  • Bias Metrics: The evaluation covers regard, AAE/WAE sentiment, individual and group fairness, and gendered word co-occurrence metrics.The metrics compare generated sentiment distributions or probability ratios across demographic conditions.
  • Results: Nucleus sampling generates more text with negative regard, while sampling produces more positive sentiment overall and decoding choices are similar for AAE/WAE sentiment.
  • Results: Beam search has relatively lower bias on IF, GF, and gendered word co-occurrence scores, followed by greedy search, top-k sampling, and nucleus sampling.
  • Diversity and Metric Interpretation: Generated-text diversity increases in the order beam, greedy, top-k, nucleus, and this diversity correlates with IF, GF, and gendered word co-occurrence scores.The paper notes that the correlation is expected from the metric formulation and motivates more comprehensive bias metrics.
Loading 2105.04054v3…