On Measures of Biases and Harms in NLP

• NLP deployment is accompanied by increasing concern about disparate impacts and harms across demographic groups.Source: §1 · p. 1
• Existing bias measures often lack explicit alignment to harms and comparative understanding of their advantages and disadvantages.Measures can differ in targeted groups, metrics, sentence structures, and definitions of bias, affecting which harms they capture.Source: §1 · p. 1
• The paper develops a theoretically motivated, empirically useful framework for describing harms and aligning bias measures with them.Case studies illustrate how a measure may conflate harms and how measures with nearly identical task definitions may capture different harms.Source: §1 · p. 1
• The authors provide documentation questions to help practitioners capture measure limitations and align operationalizations of bias with harms.The questions cover targeted demographic groups and tasks, dataset limitations, and metric definitions and motivations.Source: §1 · p. 2
• The paper organizes a tagged collection of 43 existing bias measures and their associated harms.Source: §1 · p. 2

• Bias measures are task-specific proxies that may lack context about real-world system use, and their assumptions and definitions are often unstated.Whether these measures capture meaningful and useful distinctions of bias remains an open question.Source: §2 · p. 2
• A bias measure is an evaluation standard applying one or more metrics to a dataset, with intrinsic evaluations measuring embedding associations and extrinsic evaluations measuring task decisions.Source: §2 · p. 2
• A demographic dimension is an identity axis whose instances are evaluated, often through proxies such as occupations or identity terms.Source: §2 · p. 2
• The framework distinguishes representational harms, involving harmful group representations, from allocational harms, involving disparate resource distribution between groups.Source: §2 · p. 2
• The framework assumes vulnerability to harm is mediated by a dominant–non-dominant identity-group dichotomy, inspired by but not fully aligned with Social Dominance Theory.Source: §2 · p. 4
• The paper focuses on five non-exhaustive harm types: Stereotyping, Disparagement, Dehumanization, Erasure, and Quality of Service.Source: §2 · p. 4

3 A Framework for Harms

• The framework extends existing representational-harm taxonomies with heuristics and documentation questions to align measures with five non-mutually exclusive harm categories.Delineating harms supports appraisal of mitigation strategies and comparison of trade-offs.Source: §3 · p. 4
• The authors apply harm heuristics to tag popular NLP bias measures, while noting that alternative interpretations of targeted harms are possible.Clear statements of measure motivations and relevant harms are therefore important.Source: §3 · p. 4

3.1 Harms

• Stereotyping concerns overgeneralized group-based beliefs, while Disparagement concerns model behavior reinforcing that some groups deserve less value or respect.Toxicity ratings and hate-speech detection scores are commonly associated with Disparagement.Source: §3.1 · p. 4
• Dehumanization casts disfavored groups as less than fully human, and can be difficult to measure separately from Disparagement and Stereotyping.Source: §3.1 · p. 4
• Erasure is inadequate representation of a social group, while Quality of Service harm arises when a model performs inequitably across groups.Quality of Service is quantified through performance indicators compared across commensurable groups.Source: §3.1 · pp. 4–5

3.2 Relationships between Harms

• A measure’s framing and operationalization can determine which harms it captures, so its hypothesis and design choices require careful articulation.A training-data bias may arise from multiple harms or produce multiple harms.Source: §3.2 · p. 5
• Disparagement tests can often be generated by swapping demographic identifiers, whereas stereotype tests require more culturally specific statements.Source: §3.2 · p. 5
• Dehumanization always contributes to Disparagement, while Erasure and Stereotyping can reinforce one another through missing representation of group variation.Erasure measures should use directional hypotheses and qualitative coverage rather than letting quantitative majorities determine representation.Source: §3.2 · pp. 5–6

• Bias measures are often underused or redeveloped for the same task because their motivations and design choices are poorly documented.The paper adapts existing documentation guidance for bias measures and adds sections on motivation and bias metrics.Source: §4 · p. 6
• Motivation questions ask how a measure defines bias, aligns it with harms, separates multiple harms, and specifies its groups, contexts, sources, and applications.Source: §4 · p. 6
• Dataset questions address curation methods, sources, generalizability, sentence structures, and risks from demographic proxy attributes.The guidance notes that scraped and template-generated data have different advantages and limitations.Source: §4 · p. 7
• Metric questions cover definitions and normalization, absolute versus relative evaluation, alternative measures, correlations, and whether a metric can establish absence of bias.Source: §4 · p. 7

5.1 Disparagement and Stereotyping

• Symmetric counterfactuals are generic enough to test Disparagement across groups, while asymmetric counterfactuals can capture group-specific Stereotyping; the dataset covers both.The authors note that a substitutable phrase is unlikely to express valid stereotypes unless groups share a coincidentally convergent stereotype in context.Source: §5.1 · pp. 7–8

5.2 Quality of Service, Stereotyping, and Erasure

• GAP and MAP measure Quality of Service, with MAP also probing Erasure for non-binary pronouns; occupation-based measures conflate Stereotyping and Erasure, not Quality of Service.GAP tests binary-gender pronoun-name resolution; MAP adapts GAP with gender-neutral variations.Source: §5.2 · p. 8

5.3 Dehumanization and Stereotyping

• The sentiment measure’s metric does not distinguish Dehumanization from Stereotyping; templates, word lists, and names reveal which harm an instance represents.Some sentences target stereotypes, while others examine whether a group member is socially licensed to display emotions.Source: §5.3 · p. 8

5.4 Insights from Documenting Bias Measures

• Documentation can deepen analysis over time: the regard metric lacked an explicit bias definition, and later comparisons made periodic review useful.The authors recommend treating documentation as continuous and revisiting its questions regularly.Source: §5.4 · pp. 8–9
• For coreference, occupation-based measures target Stereotyping and Erasure, while identity-term measures examine Quality of Service and sometimes Erasure.The distinction follows from whether measures examine occupational associations or model performance on gender-neutral pronouns.Source: §5.4 · p. 9
• Task constraints shape how harms are measured: datasets largely determine targets in constrained tasks, while open-ended generation relies more on bias metrics.For coreference, dataset and motivation inspection can distinguish Stereotyping from Quality of Service; generation studies may use sentiment, regard, toxicity, and psycholinguistic norms.Source: §5.4 · p. 9

6 Conclusion

• A lack of structured understanding of harms, their operationalization, and measurement can diminish the usefulness of bias measures.The paper frames bias measures as critical for estimating and mitigating potential harms toward demographic groups.Source: §6 · p. 9
• The paper organizes a framework that defines and distinguishes harms through heuristics and documentation questions to guide bias-measure development.Source: §6 · p. 9
• The documentation template supports combining, comparing, and using measures, while revisiting their limitations and comparisons over time.Source: §6 · p. 9

7 Limitations and Ethical Considerations

• The framework is US-centric, shaped by Social Dominance Theory, and does not include cultural harms.Some definitions may apply across cultural perspectives, while others may need context-specific updates.Source: §7 · p. 9
• The framework may omit harms and cultural updates, the review may miss bias measures, and the paper does not analyze specific downstream applications.Source: §7 · p. 9
• Documentation does not ensure measure validity and may create a false sense of validity; too many questions may also deter practitioners.The authors nevertheless argue that considering the questions before designing measures supports community progress.Source: §7 · p. 9

A Harm Framework Heuristics

• Stereotyping heuristics ask whether a measure tests familiar, directional group associations that oversimplify within-group diversity and would not seem natural if a commensurable group were substituted.Source: A Harm Framework Heuristics · p. 16
• Erasure heuristics examine underrepresentation or mismatches between representation and reality, including whether a measure evaluates a group against a control group.Source: A Harm Framework Heuristics · p. 16
• Disparagement heuristics cover belittling or devaluing language, societal regard, and general toxicity not tied to an established stereotype.Source: A Harm Framework Heuristics · p. 16
• Dehumanization heuristics look for language that likens groups to nonhuman life, denies their membership in a human or national group, invokes eugenics, or justifies unequal treatment.Source: A Harm Framework Heuristics · p. 16
• Quality-of-Service heuristics ask whether a measure compares model performance across commensurable demographic groups and relates to mitigation or industry use.Source: A Harm Framework Heuristics · p. 16

B A Survey of Bias Measures for Understanding Harms

• The survey describes a non-exhaustive set of primarily English-language measures across NLP tasks and aligns them with harms in Tables 1 and 2.Source: B A Survey of Bias Measures for… · p. 17

B.1 Natural Language Understanding

• Coreference measures either compare pronoun-resolution performance across gender groups or test occupational stereotypes; existing stereotype-focused measures often address singular associations, while non-binary erasure may fit the group-disparity framing better.Source: §B.1 · p. 17
• NLI bias measures test whether models label demographic-and-association sentence pairs as neutral, allowing multiple templates and demographic groups to be assessed together (Dev et al., 2019).Source: §B.1 · p. 17
• Sentiment measures hold templates constant while changing demographic cues, treating changed classifications as stereotypical associations and evaluating groups in comparison with one another.Source: §B.1 · pp. 17–18
• UnQuover interprets statistically significant differences in predicted demographic subcategories as bias, extending comparison across values such as religious groups (Li et al., 2020).Source: §B.1 · p. 18
• Relation-extraction measures compare model performance across genders and average within relationship groups, but rely on scraped text for demographic coverage (Gaut et al., 2019).Source: §B.1 · p. 18
• Masked-language-model datasets use crowdsourced sentence pairs contrasting stereotypical and other statements, while free-form text can extend beyond a single association (Nangia et al., 2020; Nadeem et al., 2021).Source: §B.1 · p. 18
• Occupation classification and toxicity or hate-speech measures compare outcomes across demographic groups; reported concerns include binary-gender coverage and uneven toxicity detection, including higher false-positive rates for AAE.Source: §B.1 · pp. 18–19

B.2 Natural Language Generation

• NLG measures probe generation with demographic prompts and metrics such as sentiment, offensiveness, word occurrence, or translation accuracy; WinoMT is widely used for gender-bias evaluation in translation.Source: §B.2 · p. 19

C.1 Case Study #1: Documentation for WinoBias (Zhao et al., 2018)

• WinoBias defines bias as more accurate pronoun resolution for occupations dominated by a pronoun’s associated gender than for occupations not dominated by that gender, a definition that does not capture gender bias in its entirety.Source: §C.1 · p. 19
• The case study places the measure in English in the United States and considers only binary male and female gender groups.Source: §C.1 · p. 19
• Its occupation list comes from the U.S. Bureau of Labor Statistics, making it an objective but non-comprehensive U.S.-specific source whose static dataset can lose relevance as statistics change.Source: §C.1 · p. 20
• The metric is the absolute score difference between pro-stereotyped and anti-stereotyped conditions, but evaluates only entities with 40 occupations in limited templates and cannot establish an absence of model bias.Source: §C.1 · p. 20

C.2 Case Study #2: Documentation for Regard (Sheng et al., 2019)

• The case study describes regard as social perception toward a demographic group, scored as negative, neutral, or positive and aligned with representational harms such as negative stereotypes or denigration.Source: §C.2 · pp. 20–21
• Template prompts probe occupations and respect in continuation generation, but manual construction can be slow, narrow, and unrepresentative of other phrasing or creator perspectives.Source: §C.2 · p. 21
• Regard scores are compared across demographics, and regard and sentiment can correlate for some prompts yet differ greatly for others, so the cited study suggests reporting both.Source: §C.2 · p. 21
• The case study cautions that limited template number and diversity, a regard classifier, and even human evaluations can leave or introduce bias despite no observed score inequality.Source: §C.2 · pp. 21–22