Source-linked AI summary
An Empirical Study of Metrics to Measure Representational Harms in Pre-Trained Language Models
Saghar Hosseini, Hamid Palangi, Ahmed Hassan Awadallah
TL;DR
Existing metrics may not adequately conceptualize representational harms, including implicit toxicity in language models. This paper proposes a language-modeling-based metric for harms toward 13 marginalized demographics and studies 24 PTLMs. The authors report demographic variation in harms and that toxic-language detectors overlook the measured notion.
Problem
Representational harms in PTLMs include implicit negative perceptions that existing metrics may fail to conceptualize or measure in alignment with the intended construct.
Source: §1 · p. 1Method
The paper proposes a language-modeling-based metric for representational harms toward 13 marginalized demographics and empirically studies 24 PTLMs.
Source: §1 · p. 1, §3.1 · p. 3Results
PTLMs show different tendencies to manifest representational harms across marginalized demographics, and the measured toxicity notion is overlooked by existing toxic-language detection models.
Source: §4.4 · p. 7, §5 · p. 9Takeaways & Limitations
The authors describe their metric as complementary to existing representational-harm metrics because it measures a different notion of harm.
Source: §4.4 · p. 7, §5 · p. 9The evaluation dataset is English, lacks intersections such as Black women, and may reflect annotator bias despite annotators being selected from different demographics.
Source: ETHICS STATEMENT · p. 10
Abstract
from arXiv · showhide
Large-scale Pre-Trained Language Models (PTLMs) capture knowledge from massive human-written data which contains latent societal biases and toxic contents. In this paper, we leverage the primary task of PTLMs, i.e., language modeling, and propose a new metric to quantify manifested implicit representational harms in PTLMs towards 13 marginalized demographics. Using this metric, we conducted an empirical analysis of 24 widely used PTLMs. Our analysis provides insights into the correlation between the proposed metric in this work and other related metrics for representational harm. We observe that our metric correlates with most of the gender-specific metrics in the literature. Through extensive experiments, we explore the connections between PTLMs architectures and representational harms across two dimensions: depth and width of the networks. We found that prioritizing depth over width, mitigates representational harms in some PTLMs. Our code and data can be found at https://github.com/microsoft/SafeNLP.
1 INTRODUCTION
The paper addresses measurement of representational harms that existing metrics may conceptualize poorly, including implicit negative perceptions that toxic-language detectors can overlook. It defines harms toward 13 marginalized demographics and proposes a metric, then studies 24 PTLMs across demographic, metric-correlation, and architecture dimensions.
1 INTRODUCTION
- Existing metrics may not align with their conceptualization of representational harms, while implicit negative perceptions can be overlooked by toxic-language detectors.The paper focuses on harms involving microaggressions, stereotypes, or implicit hate in pre-training data that may resurface in language technologies and disadvantage marginalized groups.Source: §1 · p. 1
- The study proposes a metric for representational harms toward 13 marginalized demographics and empirically examines 24 well-known PTLMs.The analysis considers demographics, correlations with existing metrics, and network architecture.Source: §1 · p. 1
2 RELATED WORK
Prior work measures representational harms in downstream applications and in language-model representations. The paper reviews these approaches and motivates an intrinsic metric aligned with the NLG task and a toxicity conception of harm.
2 RELATED WORK
- Representational-harm metrics are categorized as extrinsic, tied to downstream applications, or intrinsic, embedded in contextual word and sentence representations.Source: §2 · p. 2
2.1 EXTRINSIC
- Coreference benchmarks WinoBias and WinoGender test gendered pronoun resolution, while NLU studies perturb benchmark examples to measure label changes.Blodgett et al. (2021) raised reliability concerns about WinoBias and WinoGender; the GLUE perturbation approach uses PANDA, and the Equity Evaluation Corpus uses templated gender and race examples.Source: §2.1 · p. 2
- NLG bias studies use prompts such as BOLD and HolisticBias, but BOLD’s Wikipedia prompts and demographic name proxies have noted limitations.BOLD prompts may be noisy and contain toxic or stereotyped content; HolisticBias contains 600 descriptor terms across 13 demographics.Source: §2.1 · p. 2
- NLG classifiers measure response harms through regard, sentiment, style, or toxicity, but may encode societal biases and reduce harms to demographic label-ratio inequalities.Source: §2.1 · p. 2
2.2 INTRINSIC
- Intrinsic datasets compare stereotypical with anti-stereotypical contexts, while concerns include reliability and whether examples contain meaningful stereotypes.Crows-Pair uses contrastive pairs; StereoSet tests inter- and intra-sentence contexts across gender, race, profession, and religion.Source: §2.2 · p. 3
- CMA probes neuron and attention-head mediation of gender bias, while WEAT extensions assess sentiment associations; the paper proposes an intrinsic toxicity metric aligned with NLG.Cao et al. (2022) emphasize alignment across demographics, harm notions, applications, and evaluation-dataset quality when relating intrinsic and extrinsic metrics.Source: §2.2 · p. 3
3 MEASUREMENT MODELING
The paper follows a measurement-modeling approach: it specifies what representational harm means, then operationalizes it through language-model likelihood comparisons on annotated harmful and benign statements. Its safety score uses toxicity-weighted pairwise comparisons, with adjustments for model types where perplexity is not defined.
3 MEASUREMENT MODELING
- Measurement modeling separates conceptualization of the entity being measured from operationalization of how it is measured.Source: §3 · p. 3
3.1 CONCEPTUALIZATION
- Representational harms are defined as systematic associations of marginalized groups with negative perceptions and stereotypes in PTLMs, for 13 U.S.-context target demographics.Source: §3.1 · p. 3
3.2 OPERATIONALIZATION
- The operationalization compares language-model likelihoods for implicitly harmful and benign statements without using a majority-group power comparison.The paper notes doubts about reaching a consensual definition of majority and minority groups.Source: §3.2 · pp. 3–4
3.2.1 DATASET
- The evaluation set contains 6,541 human-annotated English sentences from ToxiGen, selected for annotator agreement on target demographics.The dataset includes implicitly harmful and benign statements toward 13 marginalized demographics; Figure 1 depicts their distributions.Source: §3.2.1 · p. 4, Figure 1 · p. 4
3.2.2 METRIC
- The metric scales each sentence’s perplexity by its toxicity score, then uses a Mann–Whitney U test to compare harmful and benign statement distributions.For auto-encoder PTLMs, the paper uses pseudo-perplexity because perplexity is not well-defined for bidirectional models.Source: §3.2.2 · p. 5
- The safety score is the effect size of the U statistic: 1 means harmful sentences have higher scaled perplexity than benign ones, while 0 indicates the reverse ordering.The U statistic uses pairwise rankings of harmful and benign sentence perplexities, weighted by their toxicity scores.Source: §3.2.2 · p. 5
4 RESULTS AND DISCUSSION
The results assess safety scores across demographics, validate the evaluation sentences’ likelihood under PTLMs, and compare scores with other harm metrics and model architectures. Distilled-model experiments find different safety outcomes for GPT-2 and BERT, with depth favored over width in auto-encoder models.
4.1 EXPERIMENT SETUP
- Safety scores were calculated for 13 marginalized demographics using 24 widely used PTLMs.The scores are reported in Table 1.Source: §4.1 · p. 5, Table 1 · p. 6
4.2 LANGUAGE MODELING
- Most PTLMs were in a reasonable log-perplexity range on the evaluation sentences; GPT-2-xl’s mean of 2.9 was comparable to WikiText, supporting that both benign and harmful statements were likely generatable.Higher log-perplexity indicates a lower chance of generation; auto-encoder models such as BERT usually have lower scores due to bidirectional architecture.Source: §4.2 · p. 5, Table 2 · p. 6
4.3 REPRESENTATIONAL HARMS TOWARDS MARGINALIZED DEMOGRAPHICS
- Safety scores for Asian, African American, Chinese, and Jewish groups were below 0.5; PTLMs were generally less likely to embed harmful contents for these groups than for other demographics.Source: §4.3 · p. 6
4.4 CORRELATION BETWEEN REPRESENTATIONAL HARMS METRICS
- Safety scores correlated negatively with CMA and SS in auto-encoders, positively with CMA and Regard in auto-regressive models, and weakly or negatively with toxicity detectors, suggesting complementary coverage.The authors note that Regard is an automatic classifier prone to manifesting representational harms, and that safety scores correlate with CMA in auto-regressive models if perplexities are accurate.Source: §4.4 · pp. 6–7, Table 3 · p. 7, Table 4 · p. 7
4.5 SAFETY SCORES ON IMPLICIT HATE SPEECH DATASET
- The safety score measured on ToxiGen and a balanced Implicit Hate subset had a correlation of 0.68, demonstrating an almost linear relationship between the dataset-based scores.The subset’s neutral examples were downsampled to balance harmful and benign examples; toxicity values were assigned for comparability.Source: §4.5 · p. 7
4.6 EFFECT OF DEPTH AND WIDTH OF THE NETWORK ON SAFETY SCORE
- Across most PTLM families, average safety scores decreased as model size grew; auto-encoder scores were more negatively correlated with width than depth, while GPT-2 showed roughly similar negative correlations with both.The authors hypothesize that more parameters increase capacity to memorize implicit toxicity in pre-training data; weight sharing may explain some architecture patterns.Source: §4.6 · p. 8, Table 5 · p. 8
4.7 SAFETY SCORE IN DISTILLED MODELS
- Distilled GPT-2 models with reasonable language-modeling performance had better safety scores than their teacher, while distilled BERT showed no significant safety improvement; auto-encoder models favor depth over width for safety and NLU performance.Reducing hidden dimension had a greater negative impact on language modeling and a positive impact on safety score.Source: §4.7 · p. 9
5 CONCLUSION
The study finds that PTLMs vary in their tendency to manifest representational harms across demographics, and that its metric captures a notion of harm overlooked by existing toxic-language detectors. Its architecture findings distinguish auto-encoder models, where depth is prioritized, from auto-regressive models, where greater width is associated with more implicit hate.
5 CONCLUSION
- The metric captures a notion of representational harm that existing toxic-language detectors overlook, and the study presents it as complementary to existing metrics.The authors report that PTLMs show higher tendencies to manifest harms toward some marginalized demographics than others, including groups less studied in prior work.Source: §5 · p. 9
- In auto-encoder models, the authors recommend prioritizing depth over width for better downstream NLU performance and reduced representational harms.For auto-regressive models, more depth does not hurt the safety score, while greater width is associated with greater capability to manifest implicit hate.Source: §5 · p. 9
- The safety score could be used as an objective function to reduce implicit hate, and the evaluation dataset could be expanded with intersectional examples.The authors give Middle Eastern women as an example of an intersection of marginalized demographics to add.Source: §5 · p. 9
ETHICS STATEMENT
The paper's evaluation dataset is synthetic, human-verified, English-language, and does not cover intersections such as Black women; annotator bias may remain. The authors also caution that the metric may not capture the full complexity of representational harms or transfer from language modeling to NLU tasks.
ETHICS STATEMENT
- The dataset is English-language and omits intersections such as Black women; annotator bias may persist despite recruiting crowd-workers from different demographics.The dataset was generated using GPT-3 and verified by a human annotator.Source: ETHICS STATEMENT · p. 10
- Because representational harms are context-dependent, changing, and human-centric, the metric may fail to capture their full complexity.The authors recommend a multidisciplinary approach involving social sciences and people in measuring and reducing representational harms.Source: ETHICS STATEMENT · p. 10
- The safety score is based on language modeling, while representational harms are task-dependent and may not transfer to NLU tasks.The authors state that harms need to be measured in relation to downstream tasks.Source: ETHICS STATEMENT · p. 10
A.1 LANGUAGE MODELING
The appendix reports mean and standard deviation of log-perplexity for benign and harmful evaluation statements. For auto-encoder models, it reports log-pseudo-perplexity.
A.1 LANGUAGE MODELING
- Mean and standard deviation of log-perplexity are reported for benign and harmful evaluation statements; auto-encoder models use log-pseudo-perplexity.Table 9 presents these values for PTLMs.Source: §A.1 · p. 13, Table 9 · p. 14
A.2 SAFETY SCORES ON IMPLICIT HATE SPEECH DATASET
The safety scores measured using ToxiGen and ImplicitHate have a reported correlation of 0.68. The ImplicitHate comparison uses balanced harmful and benign examples and assigns category-level toxicity scores for comparability.
A.2 SAFETY SCORES ON IMPLICIT HATE SPEECH DATASET
- 0.68 correlation between safety scores measured using ToxiGen and ImplicitHate demonstrates an almost linear correlation between the metrics.The reported scores cover 24 PTLMs using ImplicitHate.Source: §A.2 · p. 14
- The ImplicitHate subset contains implicit-hate or neutral examples, with neutral examples down-sampled to balance the harmful and benign sets.Because ImplicitHate lacks target-demographic and toxicity-level information, the authors assign toxicity 1 to benign examples and 2.25 to harmful examples based on average category scores in ToxiGen.Source: §A.2 · p. 14
A.3 REGARD SCORES
Regard scores count positive and neutral classifier predictions; the section reports their distribution across marginalized demographics and safety scores derived from Regard.
A.3 REGARD SCORES
- A Regard score is the percentage of positive and neutral predictions by the Regard classifier.Source: §A.3 · p. 14
- The distribution of Regard scores across 24 PTLMs is shown for each marginalized demographic.Source: §A.3 · p. 14
- Table 11 reports Regard scores, while Table 12 gives safety scores based on Regard classifier predictions.Source: §A.3 · p. 14
A.4 PRE-TRAINED LANGUAGE MODELS PARAMETERS
The appendix reports PTLM architectural dimensions alongside average safety scores and describes GPT-2 student pre-training, distillation, and few-shot evaluation across language-modeling benchmarks.
A.4 PRE-TRAINED LANGUAGE MODELS PARAMETERS
- Table 13 lists each PTLM’s layers, attention heads, and hidden dimension alongside its average safety score.Source: §A.4 · p. 14, Table 13 · p. 16
A.5 GPT-2 PRE-TRAINING AND DISTILLATION
- Twenty-three miniature GPT-2 models were pre-trained on OpenWebText for one epoch using GPT-2 pre-training hyperparameters and vocabulary (average language-modeling loss 10), then distilled from GPT-2 for one epoch over OpenWebText.All students shared hyperparameters and differed only in architecture.Source: §A.5 · p. 14
- The student models were fine-tuned with 500 examples on four language-modeling benchmarks, with perplexities reported per benchmark and averaged across benchmarks.Table 14 covers test-set perplexities, and Table 15 reports averages for distilled GPT-2 models with a GPT-2 teacher.Source: §A.5 · pp. 14, 16, Table 14 · p. 17, Table 15 · p. 17