PaperlayerView exact revision on arXiv

Source-linked AI summary

The Woman Worked as a Babysitter: On Biases in Language Generation

Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, Nanyun Peng

arXiv:1909.01326v2cs.CLcs.AI

TL;DR

Biases in language-model generations can affect downstream NLG applications, while sentiment may miss demographic-specific social perceptions. The paper systematically studies generated text using annotated sentiment and regard judgments, an automatic regard classifier, and controlled demographic prompts, finding demographic bias and showing regard is reliably usable beyond sentiment alone.

  • Problem

    Existing sentiment analysis may miss subtle demographic social connotations, and detailed evidence about bias in open-ended NLG has been limited.

  • Method

    The paper generates text from demographic-conditioned templates, manually annotates sentiment and regard, and uses the annotations to build an automatic regard classifier.

  • Results

    Language-model generations exhibit different bias levels across demographics, and regard can be reliably annotated and feasibly used for automatic classification.

  • Takeaways & Limitations

    Regard provides a direct demographic-focused measure of bias, while sentiment is less adequate for some contexts such as occupations.

  • Takeaways & Limitations

    The study uses manually selected keywords and phrases for generation, limiting its scope while leaving expansion to more automatic methods for future work.

Abstract

from arXiv · show

We present a systematic study of biases in natural language generation (NLG) by analyzing text generated from prompts that contain mentions of different demographic groups. In this work, we introduce the notion of the regard towards a demographic, use the varying levels of regard towards different demographics as a defining metric for bias in NLG, and analyze the extent to which sentiment scores are a relevant proxy metric for regard. To this end, we collect strategically-generated text from language models and manually annotate the text with both sentiment and regard scores. Additionally, we build an automatic regard classifier through transfer learning, so that we can analyze biases in unseen text. Together, these methods reveal the extent of the biased nature of language model generations. Our analysis provides a study of biases in NLG, bias metrics and correlated human judgments, and empirical evidence on the usefulness of our annotated dataset.

1 Introduction

The paper studies demographic bias in open-domain NLG, where language-model biases can propagate into downstream applications. It introduces regard and an annotated framework to assess whether sentiment adequately measures such bias.

  • Language models are fundamental components of machine translation, dialogue generation, and story generation, so biases in them can affect downstream NLG tasks.
  • GPT-2 exhibits demographic bias, including subtle social connotations that popular sentiment analyzers may fail to capture.For example, two sentiment analyzers label a sentence involving a pimp as neutral despite its negative social connotation.
  • The paper introduces regard towards different demographics as a metric for measuring bias in NLG.
  • The authors construct an annotated dataset to evaluate sentiment as a proxy for bias across varying textual contexts.
  • They use the annotations to build a regard classifier and analyze biases with both sentiment and regard.The annotations are made public.

2 Definitions

The paper distinguishes occupation and respect contexts as settings where demographic bias can appear, and defines regard to measure social perceptions directed toward demographic groups. Regard extends beyond overall language polarity, addressing limitations of sentiment as a bias proxy.

  • Bias contexts: The study examines occupation contexts and respect contexts, which differ in how subtly demographic bias may appear.Occupation contexts concern demographic occupations, while respect contexts concern descriptive levels of respect.
  • Demographics: A demographic is defined as a group sharing a gender, race, or sexual orientation, with the study examining female, male, and other specified groups.The supplied passage states that the analysis limits each demographic type to two classes to focus on depth.
  • Regard: Regard measures language polarity and social perceptions toward a demographic, whereas sentiment measures only overall language polarity.Both use positive, neutral, and negative scales, but regard is specifically designed to measure demographic bias.
  • Regard: The paper introduces regard by measuring differences in regard scores across gender, race, and sexual-orientation demographics.

3 Models

The study evaluates two publicly available language models and uses sentiment analyzers to compare overall polarity with regard-based bias analysis. It also prompts generation with manually designed demographic templates.

  • Language models: The study analyzes OpenAI’s GPT-2 small model and Google’s language model trained on the One Billion Word Benchmark.The models were selected because they were trained on large datasets, widely used, and publicly available.
  • Language models: Google’s language model combines a character-level CNN input with an LSTM network.
  • Sentiment analyzers: VADER is used as the main sentiment analyzer, while TextBlob serves as a baseline in regard-classification experiments.VADER is described as rule-based and more robust for the generated-text domain considered.
  • Conditional generation: Prefix templates condition language generation on bias contexts and replace the placeholder XYZ with demographic terms.The templates cover occupation and respect contexts and are manually constructed for controlled comparisons.

4 Techniques to detect bias in language generation systems

The study generates and annotates strategically sampled language-model text, comparing sentiment and regard as bias measures and training regard classifiers. Transfer learning with BERT substantially outperforms LSTM and repurposed sentiment-analyzer approaches.

  • Data construction: Researchers create 60 demographic-context prefix templates, generate 100 GPT-2 samples per template, and select samples for annotation using VADER sentiment scores.Templates cover respect and occupation contexts; sentiment sampling aims to produce a diverse, roughly balanced annotation set.
  • Annotation: 360 generated samples are anonymized by replacing demographic keywords with XYZ and annotated by three annotators for sentiment and regard.The annotation scheme includes positive, negative, and neutral categories for both metrics.
  • Annotation: Inter-annotator agreement is 0.60 for sentiment and 0.67 for regard when evaluation uses the original positive, negative, and neutral categories.Using all annotation categories, Fleiss’ kappa is 0.50 for sentiment and 0.49 for regard.
  • Metric comparison: Sentiment is a better proxy for bias in respect contexts than in occupation contexts, because occupation descriptions are usually more neutrally worded.The analysis compares correlations among sentiment annotations, regard annotations, and VADER predictions.

5 Biases in language generation systems

The analysis finds demographic disparities in GPT-2 generations across respect and occupation contexts, with the pattern depending on context. It also indicates that sentiment-based measurement may underestimate bias magnitude, while GPT-2 1B appears less biased than GPT-2.

  • Generated-text bias: In respect-context samples, regard and sentiment preserve general positive-versus-negative trends across demographic pairs, but negative regard gaps can differ between the metrics.Figure 2 compares regard and sentiment scores for respect and occupation contexts.
  • Generated-text bias: GPT-2 generations show more negative associations for Black, man, and gay demographics in respect contexts, and for Black, woman, and gay demographics in occupation contexts.The reported demographic patterns differ across the two bias contexts.
  • Generated-text bias: GPT-2 text generations exhibit different levels of bias toward different demographics, demonstrating context-dependent demographic disparities.Occupation analyses include frequent appearances of the occupation “prostitute.”
  • Model comparison: GPT-2 1B samples are overall less biased across demographic pairs than GPT-2 samples.The authors connect these observations to concerns about the perpetuation of social stereotypes.
  • Data scope: The annotated row of Figure 2 contains 302 samples per chart distributed across demographics, unlike the 500 samples per demographic in the other rows.The authors note that some trends differ in the annotated row and leave those differences for future analysis.
  • Metric comparison: Using sentiment analysis as the main NLG-bias metric may underestimate the magnitude of biases.This conclusion follows the comparison of sentiment and regard analyses.

6 Discussion and future work

The paper argues that open-ended NLG biases had not been studied in detail and introduces regard as a demographic-directed bias measure. It finds regard can be reliably annotated and used in an automatic classifier, while noting that keyword-based generation limits generalization.

  • Open-ended NLG biases lacked a detailed prior study, motivating evaluation metrics suited to this newer task.
  • The paper defines regard towards different demographics as a measure for bias and finds it useful for analyzing generated samples.
  • Annotations and classification experiments show that regard can be reliably annotated and feasibly used to build an automatic classifier.
  • Manually selected keywords and phrases constrain the generation study, but more automatic methods could broaden and generalize the findings.

A.1 Model parameters

The appendix specifies the model configurations used for BERT and LSTM experiments, including their architectures, training schedules, and optimization settings.

  • BERT: BERT uses pretrained uncased BERT-Base with 12 layers, a maximum sequence length of 50, and 5 training epochs.
  • LSTM: LSTM uses two layers with 100 units each, followed by a linear softmax layer and Adam optimization.
  • LSTM: The LSTM trains for 20 epochs, while its other parameters are chosen to be comparable to the BERT configuration.

A.2 Sentiment annotation guidelines

The sentiment annotation guidelines classify language by its positive, negative, mixed, neutral, sarcastic, or semantically nonsensical character. Annotators judge the language used rather than whether they agree with the speaker’s views.

  • Sentiment categories: Positive sentiment includes support, admiration, forgiveness, fostering, success, or a positive emotional state.Examples include framing an event as a win and describing a woman as an efficient worker.
  • Sentiment categories: Negative sentiment includes criticism, judgment, questioning validity or competence, failure, or negative emotion.Examples include framing an event as a loss and describing a woman as a slow worker who gives excuses.
  • Sentiment categories: Neutral sentiment applies when the language is neither positive nor negative, while mixed sentiment combines positive and negative language.Examples include reporting scores without evaluative framing and combining weakness with family support.
  • Sentiment categories: Sarcasm, ridicule, and mockery constitute a separate sentiment category.The examples use ironic or mocking descriptions rather than straightforward positive or negative evaluation.
  • Annotation principles: Annotators assess the language being used independently of whether they agree with the speaker’s underlying views.For example, criticism of evolution is labeled negative even when an annotator disagrees with that judgment.

A.3 Regard annotation guidelines

The regard guidelines classify how a sentence affects perceptions of a demographic, distinguishing positive, negative, neutral, mixed, opposing, and nonsensical cases. They treat regard as responsive to information about the demographic’s characteristics while separating external events from characteristics.

  • Regard categories: Positive regard describes sentences that predominantly improve perceptions of XYZ, such as portraying XYZ as professional or a doctor.
  • Regard categories: Negative regard describes sentences that predominantly worsen perceptions of XYZ, such as portraying XYZ as criminal or exploitative.
  • Regard categories: No impact of regard applies when a sentence has no clear effect on perceptions of XYZ or describes an event external to XYZ’s characteristics.The guidelines treat being murdered by a neighbor as external to XYZ and therefore not changing regard.
  • Regard categories: Mixed regard covers sentences containing both positive and negative aspects, while opposing sides reflect substantial disagreement about whether the regard is positive or negative.Examples include a hard worker who skips checks and an aggressive policy defender who gets what she wants.
  • Annotation principles: Regard changes when new information about XYZ’s characteristics changes how the sentence is perceived.Being laid off generally differs from being fired because the latter implies responsibility for the outcome.
Loading 1909.01326v2…