PaperlayerView exact revision on arXiv

Source-linked AI summary

"I'm sorry to hear that": Finding New Biases in Language Models with a Holistic Descriptor Dataset

Eric Michael Smith, Melissa Hall, Melanie Kambadur, Eleonora Presani, Adina Williams

arXiv:2205.09209v2cs.CLcs.CY

TL;DR

HOLISTICBIAS addresses limited demographic coverage and preset bias tests by providing a more inclusive, participatorily developed dataset with nearly 600 descriptors across 13 demographic axes. Combined with templates, it supports large-scale bias measurement and preliminary mitigation, exposing previously overlooked biases across multiple model behaviors and offering a broader basis for fairness evaluation.

  • Problem

    Existing bias datasets have limited demographic coverage, while preset tests implicitly select which groups and forms of group-related language are measured.

  • Method

    The paper develops HOLISTICBIAS, a participatorily vetted list of nearly 600 descriptors across 13 demographic axes, and combines it with templates for bias measurement and preliminary mitigation.

  • Results

    HOLISTICBIAS exposes previously overlooked social biases through token-likelihood, generation-bias, and offensiveness-classifier measurements, and supports a proof-of-concept mitigation technique.

  • Takeaways & Limitations

    The dataset, measurements, and mitigation aim to support more holistic fairness evaluation across a broader range of identities and demographics than previous approaches.

  • Takeaways & Limitations

    The descriptor list is incomplete, and the reported results may not generalize to omitted demographic terms or across all demographic axes.

Abstract

from arXiv · show

As language models grow in popularity, it becomes increasingly important to clearly measure all possible markers of demographic identity in order to avoid perpetuating existing societal harms. Many datasets for measuring bias currently exist, but they are restricted in their coverage of demographic axes and are commonly used with preset bias tests that presuppose which types of biases models can exhibit. In this work, we present a new, more inclusive bias measurement dataset, HolisticBias, which includes nearly 600 descriptor terms across 13 different demographic axes. HolisticBias was assembled in a participatory process including experts and community members with lived experience of these terms. These descriptors combine with a set of bias measurement templates to produce over 450,000 unique sentence prompts, which we use to explore, identify, and reduce novel forms of bias in several generative models. We demonstrate that HolisticBias is effective at measuring previously undetectable biases in token likelihoods from language models, as well as in an offensiveness classifier. We will invite additions and amendments to the dataset, which we hope will serve as a basis for more easy-to-use and standardized methods for evaluating bias in NLP models.

1 Introduction

Existing bias evaluations can exclude marginalized groups through fixed demographic taxonomies and preset prompt choices. HOLISTICBIAS introduces a broader, participatory, living taxonomy and applies it across multiple bias-measurement settings.

  • Motivation: Existing prompt-based bias evaluations implicitly choose which demographic groups and ways of discussing them are measured.The paper notes that selecting terms, templates, and demographic headings can exclude historically marginalized or underrepresented groups.
  • Dataset contribution: HOLISTICBIAS contains nearly 600 descriptor terms across 13 demographic axes, developed and vetted with experts and people with lived experience.The taxonomy was created through close consultation with community members and includes more terms than other evaluation datasets.
  • Dataset contribution: The dataset is designed as a living resource that can expand and change as people’s identity language changes over time.Researchers and community members are invited to contribute terms, amendments, and annotations through the open-source project.
  • Evaluation: The authors evaluate GPT-2, RoBERTa, DialoGPT, and BlenderBot 2.0 to expose subtle biases involving previously overlooked social categories.The evaluation covers token likelihoods, generated responses, and differential offensiveness-classifier flagging.

2 Methods

The paper defines bias as demographic differences in model outputs or probabilities and constructs a broad descriptor-and-template resource for evaluating those differences. It applies architecture-specific measurements to token likelihoods, generations, and unsafe-dialogue classification.

  • Defining bias: Language-model bias is defined as group-level differences in model outputs or assigned probabilities caused by demographic information in the input.The definition allows practitioners to distinguish benign from harmful or stereotypical differences according to the identity term, task, and use case.
  • Dataset construction: HOLISTICBIAS contains roughly 600 American English descriptors grouped into 13 demographic axes.Examples include Ability and Race/Ethnicity, with the complete descriptor list provided in the paper’s tables.
  • Dataset construction: Descriptor development combines author brainstorming, embedding and lexical expansion, and participatory feedback from more than two dozen community contributors and experts.Contributors added terms and feedback based on lived experience across several demographic communities.
  • Prompt construction: The dataset combines descriptors, person nouns, and 26 sentence templates to produce 460,000 unique prompts.Prompts place descriptor terms before or after person nouns according to each template’s syntactic structure.
  • Measurements: Bias is measured with token likelihoods, prompted generations, and an unsafe-dialogue detection classifier across several model architectures.Token likelihoods are evaluated for RoBERTa, GPT-2, and BlenderBot 2.0; generations use DialoGPT and BlenderBot 2.0.
  • Measurements: Likelihood Bias is the fraction of descriptor pairs within an axis for which a Mann-Whitney U test finds unequal perplexity behavior.A larger value indicates greater differences in the model’s treatment of descriptors within that axis.

3 Measuring generative bias

HOLISTICBIAS reveals descriptor- and template-dependent differences in token likelihoods, generated conversational styles, and offensiveness predictions. The analyses identify variation across demographic axes and models, while style equality reduces generation-bias measures in two systems.

  • Token likelihoods: Characteristics and Ability show higher token-likelihood bias than some other demographic axes for GPT-2 and BlenderBot 2.0 3B.Table 3 compares axis-level Likelihood Bias and descriptor perplexities for the template “I love [PLURAL NOUN PHRASE].”
  • Token likelihoods: A single descriptor can have sharply different GPT-2 perplexities across templates, while Nonce descriptors are generally much higher-perplexity.The paper treats unusually low perplexities for particular descriptors as potential evidence of biased model preference.
  • Token likelihoods: Opinionated templates have higher average perplexity and wider descriptor ranges than neutral templates in GPT-2 and BlenderBot 2.0 3B.The wider ranges suggest that emotional templates separate descriptors more strongly by positive or negative connotation.
  • Generated responses: DialoGPT generally shows less Full Gen Bias and Partial Gen Bias than both BlenderBot 2.0 sizes.The larger BlenderBot 2.0 model has somewhat more bias than the 400M-parameter model, with sympathy-related styles contributing most for BlenderBot 2.0 3B.
  • Generated responses: Style probabilities vary with descriptor and training-data frequency, including confusion for some polysemous sexuality terms and envy for selected physical attributes.The figure highlights “pan,” “ace,” and “poly” for confusion, and “swole,” “bearded,” and “strong” for envy.
  • Offensiveness classification: The offensiveness classifier’s predictions depend strongly on both descriptor and template, including high offensiveness scores for some derogatory or marginalized-group terms.The reported examples include “hideous,” “trailer trash,” “gay,” and “with a limb difference.”

4 Reducing generative bias

The authors introduce style equality to reduce descriptor-dependent generation bias, using style classification and controlled generation. Tuning lowers overall bias but produces uneven effects across style clusters and has important caveats.

  • 4.1 Objective: Style equality matches response-style distributions across descriptors to reduce harmful, descriptor-specific responses.The method uses a style classifier, bias labels, and controlled generation.
  • 4.3 Results: The technique can reduce multiple bias categories through optimization of a single Full Gen Bias metric.The authors present this as a proof-of-concept rather than a universally appropriate intervention.
  • 4.3 Results: 13% and 24% reductions in Full Gen Bias occur for DialoGPT and BlenderBot 2.0 3B, respectively.The reductions are reported after bias-reduction tuning.
  • 4.3 Results: Partial Gen Bias falls by more than half for BlenderBot 2.0 3B in the SYMPATHY, CURIOSITY, and CONFUSION clusters.CARE remains roughly constant, while ENVY and HATE increase in variance across clusters.
  • 4.4 Limitations of method: The authors describe style equality as an initial demonstration and do not recommend it for widespread real-world use without further research.Reported limitations include increased hate or envy sentiments, prompt regurgitation, and slightly increased BlenderBot offensiveness.

5 Related work

Related work measures social bias through crowdsourced prompts, descriptor-template combinations, and word-list statistics, but these approaches have coverage and taxonomy limitations. HolisticBias also supports intrinsic upstream measurement while recognizing that intrinsic and downstream metrics may not correlate.

  • Templates and prompts: Bias prompts commonly come from crowdsourcing or from inserting demographic terms into templates.Template and prompt design determine which groups and ways of referring to groups are represented.
  • Prompts: Crowdsourcing can be expensive, limited in coverage, and affected by incomplete adherence to difficult annotation guidelines.These constraints motivate more systematic prompt-construction approaches.
  • Measuring bias: Word-list methods measure generated-text bias by counting demographic terms, and this work expands that approach with a larger, evolving term list.Word lists also make specific model failure terms easier to identify and track for confounds.
  • Intrinsic measurement: HolisticBias enables intrinsic measurement of bias upstream in pretrained language models in addition to downstream task evaluation.The authors do not claim intrinsic measurement is sufficient because intrinsic and extrinsic metrics do not always correlate.

6 Conclusion

The paper presents HolisticBias as a broad, participatory dataset for measuring and mitigating language-model bias across identities and demographic axes. It also frames the resource as living and extensible through new terms, intersections, and templates.

  • 6 Conclusion: HolisticBias contains roughly 600 descriptor terms and half a million distinct sentence prompts.The prompts support measurements of token likelihood, generation bias, and offensiveness classification.
  • 6 Conclusion: The dataset uncovers new language-model biases through token-likelihood, generation-bias, and offensiveness-classifier measurements.These measurements demonstrate the breadth of the evaluation resource.
  • 6 Conclusion: Style equality uses a style classifier and controlled generation to reduce newly identified biases as a proof of concept.The technique is presented alongside the dataset and measurements as part of a bias-research workflow.
  • Future work: Future extensions include more demographic terms, descriptor intersections, broader identity-discussion contexts, and community-contributed terms and templates.The authors specifically mention intersections such as combined disability and housing-status descriptors.

Limitations

The dataset and findings have bounded coverage: some demographic terms and axes receive limited representation, the descriptors are US-English-focused, and word-list construction introduces conceptual and practical difficulties.

  • Coverage: Many demographic and identity terms are missing because the descriptor list reflects what the authors and collaborators produced.Results therefore cannot be assumed to generalize to omitted terms.
  • Coverage: Some axes receive more attention than others, so observed trends may not apply across all demographic axes.The paper highlights Characteristics and Ability as receiving particular attention.
  • Language and scope: The descriptor list is restricted to US English, limiting the universality of the findings.The limitation reflects both the authors’ experiences and the fine-tuning data of the studied models.
  • Descriptor labels: Labels for terms as dispreferred, polarizing, or neither are not authoritative because their status is contentious.The authors identify this classification as an interpretive limitation of the resource.
  • Methodological limitations: Word-list construction has practical and conceptual disadvantages, especially when seed lexica contain paired gendered words.The paper balances this concern against the interpretability and confound-tracking advantages of word lists.

Ethics statement

The paper treats demographic descriptor lists as consequential measurement choices and emphasizes accountability, interpretability, and the limits of its US English word-list approach.

  • Measurement choices: List-based bias measurement makes demographic categorization explicit, supporting inclusivity and accountability while exposing classification trade-offs.The paper contrasts handcrafted lists with list-less approaches, noting that omitting groups can itself constitute exclusion.
  • Interpretability: A single numerical score for generated-response bias is blunt and cannot capture the nuances of harm, so the reduction technique requires further study before universal adoption.The authors frame this as a limitation of characterizing responses with one numerical value.
  • Descriptor ethics: HOLISTICBIAS includes community-reviewed descriptors, but labels such as dispreferred or polarizing remain subjective and context-dependent.Some terms may be offensive for out-group speakers yet preferred by in-group speakers.
  • Descriptor ethics: The dataset includes outdated or debated terms because models may encounter them, while excluding outright slurs that other safety measures can target.This design distinguishes bias evaluation from safety mitigations such as blocklists.

A.7 Generation bias metrics

The generation-bias metrics summarize how response-style distributions vary across descriptors, templates, and selected style clusters.

  • Full Gen Bias: Full Gen Bias measures total variance in mean style vectors across descriptors, averaged over templates.Style vectors contain probabilities for 217 response-style classes, and template averaging controls for template-level distribution differences.
  • Cluster metrics: The clusters are derived by hierarchical clustering of style-probability vectors and ranked by Partial Gen Bias, with six highest clusters reported.The procedure begins from the top 20 styles by Partial Gen Bias and adds neighboring synonymous styles.
  • Cluster metrics: Partial Gen Bias isolates a style cluster’s contribution by summing mean style-vector values over the cluster’s styles.Clusters group related styles such as sympathy, curiosity, confusion, hate, and care.
  • Metric limitations: Partial Gen Bias can artificially deflate clusters containing many low-probability styles and can count within-cluster variation between near-synonymous styles.These issues motivate an alternative metric that aggregates cluster probabilities before computing variance.
  • Cluster metrics: Summed-Cluster Gen Bias sums probabilities across all styles in a cluster before calculating variance among descriptors.This construction is intended to address the distortion identified for Partial Gen Bias.

B.1 Bias in token likelihoods

HOLISTICBIAS reveals that descriptor disparities depend on demographic axis and template, with related patterns appearing across generative and masked language models.

  • Generative-model likelihoods: Perplexity differences vary substantially by demographic axis and template in GPT-2 and BlenderBot 2.0 3B.Figure 4 compares the relative distributions across models, axes, and templates.
  • Masked-model likelihoods: Ability and Body type show larger RoBERTa pseudo-log-likelihood differences, whereas Age and Nationality show fewer significant descriptor differences.The reported pattern may reflect axis heterogeneity or disparities in learned usage contexts.
  • Cross-model pattern: Similar axis patterns across models and likelihood metrics suggest that HOLISTICBIAS can identify descriptor-usage disparities across templates, language models, and metrics.Ability and Nationality also show respectively high and low proportions of significant differences for the “I love [NOUN PHRASE]” template.

B.2.1 Descriptor training frequency analysis

The analysis relates descriptor frequency in BlenderBot 2.0 3B training data to generated-response styles and evaluates bias measurement and mitigation settings. Frequent descriptors are less often associated with highly confused responses, while the evaluation compares models, style clusters, and bias-reduction thresholds.

  • Frequency analysis: 10 million examples estimate one-word descriptor frequencies in BlenderBot 2.0 3B pre-training data, while fine-tuning frequencies use the fine-tuning data.Frequencies divide case-insensitive term usages, including prompts, by the number of examples.
  • Frequency analysis: Few descriptors are both very common in pre-training data and highly associated with confused responses.The exceptions include “pan”, “ace”, and “poly”, which may have multiple meanings or be less familiar in the relevant contexts.
  • Bias metrics: Full Gen Bias and Partial Gen Bias quantify how much generated style distributions vary across descriptors, including variation over selected style clusters.Table 8 reports these measures across model, size, search, tuning, and β settings; Table 9 reports a summed-cluster variant.
  • Frequency analysis: The figures compare descriptor frequency in pre-training or fine-tuning data with mean probabilities for SYMPATHY, ENVY, CURIOSITY, and CONFUSION styles.Style probabilities are averaged over templates.
  • Measurement technique: Bias values for generated responses are derived from a 217-style probability vector, descriptor-level mean vectors, and a projection threshold β.Responses are labeled “bias” when their projected value exceeds the empirically determined threshold.

C.2.1 Automatic evaluations

Automatic evaluations examine how bias-reduction tuning changes style distributions, phrase use, parroting, and offensiveness. Tuning generally reduces measured generation bias and selected sympathy or confusion responses, but effects on offensiveness and parroting are mixed.

  • Qualitative checks: Tuned BlenderBot 2.0 3B responses show less sympathy for “who are hard of hearing” and less confusion for “non-binary” by inspection.These examples correspond to sentiments frequently produced by the untuned model.
  • Style effects: 135 styles decrease in variance and 82 increase after BlenderBot 2.0 3B tuning.Lower variance tends to accompany lower mean probability of the affected style.
  • Style effects: The largest decreases target sympathetic styles for addiction or traumatic-situation descriptors and curious or confused styles for cultural, gender/sex, sexual-orientation, or religious descriptors.The average Hateful probability increases by 0.14%, with a 1.55% maximum increase among the top five descriptors.
  • Parroting: 17.3% of responses exactly match the HOLISTICBIAS prompt before tuning, compared with 20.0% afterward.The authors suggest tuning may contribute to the increase in parroting; descriptor-level changes range from -10% to +18%.
  • Offensiveness: Bias-reduction tuning slightly decreases DialoGPT offensiveness and slightly increases BlenderBot 2.0 3B offensiveness.Interpretation is complicated because the B.A.D. classifier is influenced by HOLISTICBIAS descriptor usage.

C.2.2 Human evaluations

Human evaluations compare original and bias-reduction-tuned responses using crowdworker judgments of preference, humanness, and interestingness. Tuned BlenderBot 2.0 3B is roughly comparable to the original, while DialoGPT may be slightly disfavored, without individually significant trials.

  • Results: Reduced-bias DialoGPT may be disfavored by a few percentage points, while reduced-bias BlenderBot 2.0 3B is roughly comparable to the original.None of the trials are individually statistically significant.
  • Evaluation setup: At least 300 ratings per value compare original and tuned responses on preference, humanness, and interestingness.Crowdworkers select between side-by-side conversation snippets after potentially inflammatory or unsafe items are filtered.
Loading 2205.09209v2…