Source-linked AI summary
BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation
Jwala Dhamala, Tony Sun, Varun Kumar, Satyapriya Krishna, Yada Pruksachatkun, Kai-Wei Chang, Rahul Gupta
TL;DR
Social biases in open-ended language generation are insufficiently benchmarked despite the models’ broad use and documented risks. BOLD provides a large Wikipedia-derived benchmark with multiple bias metrics, and experiments show most evaluated models are more socially biased than Wikipedia text. The paper therefore emphasizes benchmarking model behavior before downstream deployment.
Problem
Open-ended language generation can produce socially biased text, but systematic evidence and benchmarking for this setting remain limited despite its many applications.
Method
BOLD introduces 23,679 Wikipedia-derived prompts across five domains and evaluates generated text with metrics including toxicity, psycholinguistic norms, and gender polarity.
Results
Most evaluated language models exhibit larger social biases than human-written Wikipedia text across domains, especially toward historically disadvantaged groups.
Takeaways & Limitations
The findings highlight the importance of benchmarking open-ended generation before deploying language models in downstream tasks.
Takeaways & Limitations
BOLD covers limited demographic domains and groups, so performance on BOLD does not indicate that a model is completely fair.
Abstract
from arXiv · showhide
Recent advances in deep learning techniques have enabled machines to generate cohesive open-ended text when prompted with a sequence of words as context. While these models now empower many downstream applications from conversation bots to automatic storytelling, they have been shown to generate texts that exhibit social biases. To systematically study and benchmark social biases in open-ended language generation, we introduce the Bias in Open-Ended Language Generation Dataset (BOLD), a large-scale dataset that consists of 23,679 English text generation prompts for bias benchmarking across five domains: profession, gender, race, religion, and political ideology. We also propose new automated metrics for toxicity, psycholinguistic norms, and text gender polarity to measure social biases in open-ended text generation from multiple angles. An examination of text generated from three popular language models reveals that the majority of these models exhibit a larger social bias than human-written Wikipedia text across all domains. With these results we highlight the need to benchmark biases in open-ended language generation and caution users of language generation models on downstream tasks to be cognizant of these embedded prejudices.
1 INTRODUCTION
BOLD addresses limited evidence on social biases in open-ended language generation by using natural Wikipedia-derived prompts and evaluating models across multiple bias dimensions. Experiments show that most evaluated models exhibit greater social bias than Wikipedia text, particularly toward historically disadvantaged groups.
- Natural language generation models support applications including translation, summarization, storytelling, conversation bots, and writing assistance.They generate probable word sequences autoregressively from prompt context.
- Limited prior work systematically measures bias in open-ended generation, despite evidence that unchecked models can reinforce stereotypes and produce unfair outcomes.Bias can appear when generated negative text occurs more frequently for particular social groups.
- BOLD uses seed words matching human-written text distributions to avoid contrived prompts that may reinforce triggered biases or miss other critical biases.The approach is intended to expose bias under more natural prompting conditions.
- 23,679 unique prompts cover five domains and 43 subgroups, while proposed metrics measure toxicity, psycholinguistic norms, and gender polarity alongside existing metrics.The novel metrics were validated against crowd-worker ratings.
- Most evaluated models exhibit larger social biases than Wikipedia text, especially toward historically disadvantaged groups.CTRL-THT, CTRL-OPN, and GPT-2 generate more polar texts than BERT and CTRL-WIKI on the bias metrics.
2 RELATED WORK
Related work has quantified harmful stereotypes and group disparities across NLP tasks, but open-ended generation remains comparatively underexplored. Prior generation studies use small curated prompts or adversarial triggers, while contemporary benchmarks evaluate stereotypical sentence preferences.
- Recent NLP research exposes harmful cultural stereotypes, negative stereotyping, and inadvertent group segregation through bias measurements.
- Word-embedding and association-test approaches measure bias through gender-subspace projections or stereotype-related permutation tests.Other work constructs counterfactual datasets to compare model behavior across groups.
- Prior open-ended generation research found GPT-2 produced lower sentiment and regard scores for contexts associated with certain groups.That study used 60 manually curated prompts, while adversarial triggers were shown to control generation bias.
3 BOLD: BIAS IN OPEN-ENDED LANGUAGE GENERATION DATASET
BOLD constructs a fairness benchmark from naturally occurring English Wikipedia text, organizing prompts across demographic domains and filtering them for relevance and quality. The dataset is cleaned and anonymized before metric calculation.
- Wikipedia is used as the prompt source because its collaboratively written articles span many domains and demographics and undergo quality control.This addresses concerns that expert- or crowd-collected prompts may not reflect natural text beginnings.
- BOLD studies profession, gender, race, religious belief, and political ideology across defined demographic groups.The gender domain uses male and female actors, while the race domain includes European American, African American, Asian American, and Latino or Hispanic American groups.
- BOLD collection: For each domain, the collection process identifies relevant Wikipedia pages and extracts sentences mentioning the target profession within the first eight words.
- BOLD collection: Extracted Wikipedia text is post-processed and cleaned because pages may contain irrelevant links, incomplete text, or content unrelated to the target group attribute.The dataset also compares language-model generations with original Wikipedia sentences.
- BOLD collection: Prompts are anonymized during metric calculation by replacing person names with “Person” and target professions, religions, and ideologies with “XYZ”.This is intended to reduce evaluator or metric bias toward the studied person or group.
4 EVALUATION METRICS
The paper evaluates generated-text bias through sentiment, toxicity, regard, psycholinguistic norms, and gender-polarity metrics. These metrics capture emotional, harmful, demographic, and gender-associated signals from multiple angles.
- Sentiment, toxicity, regard, and emotion lexicons measure positive or negative feelings in texts prompted by demographic or ideological groups.Profession prompts additionally receive metrics for polarity toward male or female gender.
- VADER assigns each text a sentiment score from −1 to 1 and classifies sentiment using thresholds of ≥0.5 and ≤−0.5.
- Toxicity is assigned when a fine-tuned BERT classifier labels text as toxic, severe toxic, threat, obscene, insult, or identity threat.
- Psycholinguistic norms extend expert-rated word lexicons and aggregate them into text-level emotion measures across VAD and BE5 dimensions.VAD uses Valence, Arousal, and Dominance; BE5 uses Joy, Anger, Sadness, Fear, and Disgust.
- Gender polarity: The unigram gender-polarity metric compares counts of male- and female-specific tokens, labeling texts neutral when both counts are zero.It captures direct gendered words but not indirectly gender-associated words.
- Gender polarity: The embedding-based gender metric projects words onto the gender direction and aggregates word scores using Gender-Wavg or Gender-Max.Gender-Wavg uses magnitude-weighted averaging, whereas Gender-Max selects the most gender-polar word.
- Gender polarity: Global gender-polarity scores classify text as male at ≤−0.25 and female at ≥0.25.The thresholds were determined empirically from texts with known gender labels.
5 GENERATING WITH LANGUAGE MODELS
The study prompts several language models with BOLD seed-word sequences to compare models differing in architecture, training strategy, and training corpora. It evaluates BERT, GPT-2, and CTRL variants using Wikipedia, Thought, and Opinion control codes.
- BOLD prompts trigger language models to generate text from sequences of seed words, enabling comparison across models with different training strategies and corpora.
- BERT: BERT uses bidirectional context and is pretrained on English Wikipedia and BooksCorpus; here it fills in words after Wikipedia-derived seed prompts.
- GPT-2: GPT-2 predicts the next word autoregressively and was pretrained on WebText collected from filtered web pages, including Reddit sources.
- CTRL: CTRL conditions generation on control codes that govern style, content, and task-specific behavior.
- CTRL: The experiments use CTRL-WIKI, CTRL-THT, and CTRL-OPN, corresponding to Wikipedia, Thought, and Opinion control codes.
- CTRL: The Wikipedia control code traces to English Wikipedia, while Thought and Opinion codes trace to r/showerthoughts and r/changemyviews, respectively.
6 EXPERIMENTS
Experiments compare generated texts from multiple language models with Wikipedia text across demographic domains using automated bias metrics, with human-annotation validation. The results reveal recurring differences in gender polarity, sentiment, regard, toxicity, and psycholinguistic norms across groups and models.
- Gender polarity metrics: Only 6.57% of profession texts were classified as male or female, while female classifications were higher for healthcare and medicine and male classifications dominated most other profession groups.This pattern held across Gender-Max, Gender-Wavg, and unigram matching metrics and across data sources.
- Sentiments: 76.72% of gender-domain texts had neutral sentiment; female prompts produced fewer negative-sentiment texts than male prompts, with 0.047 versus 0.069 respectively.Female positive sentiment was 0.17763 versus 0.17041 for male prompts, but this difference was not significant; the negative-sentiment difference had p-value<0.01.
- Psycholinguistic norms: Male prompts generated more texts containing anger, sadness, fear, and disgust, whereas female prompts generated more texts containing joy and dominance across all language models.The psycholinguistic-norm pattern was consistent with the sentiment results showing fewer negative-sentiment texts for female prompts.
- Race: African American prompts had the largest negative sentiment and toxicity proportions among racial groups, with negative sentiment 0.08154 and toxicity 0.00297.Both differences across groups had chi-square test p-value < 0.001, and positive regard for European Americans was equal or larger in five of six models.
- Political ideologies: Negative sentiment was highest for fascism across political ideologies, although positive sentiment was not lowest for fascism across all models.The authors identify this pattern as undesirable and discuss handling generations associated with extremist ideologies appropriately.
- Gender polarity metrics: The average male-to-female ratios were 2.29 for Wikipedia, 1.25 for BERT, 3.18 for GPT-2, 2.94 for CTRL-WIKI, 1.85 for CTRL-OPN, and 2.15 for CTRL-THT.GPT-2 had the largest ratio, while BERT had the smallest.
- Toxicity: Less than 0.5% of texts were classified as toxic across all data sources and domains, while CTRL-OPN, CTRL-THT, and GPT-2 exceeded Wikipedia, BERT, and CTRL-WIKI in toxic-text proportions.The authors suggest low toxicity may reflect weak triggering of highly polar text or mismatch between the toxicity classifier's social-media training data and BOLD.
- Comparison of language generation models: CTRL-THT, CTRL-OPN, and GPT-2 generated more polar texts across gender polarity, sentiment, toxicity, and regard than BERT and CTRL-WIKI.The paper links this difference to training-data sources: social-media data for CTRL-THT and CTRL-OPN versus Wikipedia data for CTRL-WIKI and BERT.
7 LIMITATIONS AND DISCUSSIONS
The paper identifies scope, metric, annotation, and baseline limitations that constrain how BOLD results should be interpreted. BOLD covers only selected groups and domains, while Wikipedia and human annotations are not unbiased ground truths.
- Scope limitations: BOLD covers selected demographic domains and groups, not the full spectrum of real-world diversity.Gender is binary, and race includes a small subset conceptualized within American culture.
- Scope limitations: BOLD results indicate bias in the covered categories but do not establish that a model is completely fair.Additional domains, groups, languages, social biases, and cultural contexts remain outside the benchmark’s scope.
- Measurement limitations: The study’s automated metrics cannot fully capture social-bias degree, despite alignment with human judgments for sentiment, toxicity, and gender polarity.Crowd-worker annotations are also influenced by annotator cultural background and demographics.
- Baseline limitation: Wikipedia is not a fair bias baseline because its authorship distribution is highly skewed and its text exhibits biases resembling generated text.The authors therefore caution against treating Wikipedia comparisons as comparisons against unbiased text.
8 CONCLUSION
The conclusion presents BOLD and its metrics as a test bed for fairness in open-ended language generation. Experiments find that language models more often generate negative group- or idea-related connotations, with some models conforming more strongly to social biases.
- Contributions: BOLD is a dataset and metric suite for evaluating fairness in open-ended language generation.The resources are intended as a test bed for researchers and practitioners benchmarking language models.
- Findings: Language models more frequently generate negative sentiments and toxicity toward African American people and male words in profession contexts.These are examples reported from the experiments across the studied demographic domains.
- Findings: GPT-2, CTRL-THT, and CTRL-OPN conform more to social biases than BERT and CTRL-WIKI.The comparison concerns the models evaluated in the paper.
- Implication: The findings motivate studying and benchmarking social biases in open-ended generation to help prevent reinforcing detrimental biases downstream.The paper frames BOLD as a resource for researchers and practitioners.
A.1 Data Collection Details
The appendix states that the authors used specified pages to collect BOLD data, without detailing those pages in the supplied passage.
- Data sources: The authors used specified pages as the sources for collecting BOLD data.The supplied appendix passage does not identify the pages themselves.
- Data sources: BOLD data collection relied on pages selected by the authors.The passage describes the collection source at a high level only.
- Data sources: The appendix refers readers to the following pages for BOLD’s data-collection details.No page list is included in the supplied text.
A.2 Implementation Details
The appendix records implementation settings for BERT, GPT-2, and CTRL text generation. It also states that the experiments use specified hyperparameters and implementation details.
- General settings: The authors specify hyperparameters for text generation experiments.The appendix introduces the settings used for the generation procedures.
- BERT: BERT uses bert-large-cased with maximum sentence length 15, temperature 0.7, burn-in 200 iterations, and maximum 500 iterations.The implementation is provided through the cited BERT text-generation implementation.
- GPT-2: GPT-2 uses top-k 40 and top-p 0.95 sampling to create natural, coherent sentences.The passage links this combination to recommendations from prior work.
- CTRL: CTRL uses HuggingFace’s default parameters with repetition penalty 1.2 and top-p 0.9.These settings are stated for the CTRL experiments.
A.2.4 VADER.
The section specifies VADER-based sentiment scoring and the annotation guidelines used to classify generated texts by sentiment, toxicity, and gender.
- VADER sentiment scores are computed with the implementation’s default parameters.
- Sentiment: Annotators choose the primary sentiment, selecting the stronger emotion when a text contains mixed sentiments.
- Sentiment: Positive, negative, and neutral sentiment labels are illustrated with generated-text examples, with neutral text defined as lacking strong positive or negative emotions.