Source-linked AI summary
Challenges in Measuring Bias via Open-Ended Language Generation
Afra Feyza Akyürek, Muhammed Yusuf Kocyigit, Sejin Paik, Derry Wijaya
TL;DR
Bias measurements from open-ended generation may vary with experimental choices, complicating efforts to assess models. This paper analyzes those choices and finds they can yield conflicting conclusions, motivating more comprehensive reporting of measurements.
Problem
Researchers use open-ended text generation to measure social-group bias, but the resulting conclusions may vary with experimental settings.
Source: §1 · pp. 1–2Method
Results
Takeaways & Limitations
The authors recommend more comprehensive reporting across prompts, metrics, tools, and representative decoding settings, with human evaluation on a subsample of generations.
Source: §5 · pp. 5–6The authors frame their concern around ad-hoc settings in generation-based bias studies, where changing a parameter such as temperature can produce inconsistent interpretations.
Source: §6 · p. 6
Abstract
from arXiv · showhide
Researchers have devised numerous ways to quantify social biases vested in pretrained language models. As some language models are capable of generating coherent completions given a set of textual prompts, several prompting datasets have been proposed to measure biases between social groups -- posing language generation as a way of identifying biases. In this opinion paper, we analyze how specific choices of prompt sets, metrics, automatic tools and sampling strategies affect bias results. We find out that the practice of measuring biases through text completion is prone to yielding contradicting results under different experiment settings. We additionally provide recommendations for reporting biases in open-ended language generation for a more complete outlook of biases exhibited by a given language model. Code to reproduce the results is released under https://github.com/feyzaakyurek/bias-textgen.
1 Introduction
Open-ended generation makes it possible to test how language models treat social groups, but bias measurements can depend heavily on experimental design. The paper examines choices throughout this measurement pipeline and finds they can produce conflicting bias conclusions.
1 Introduction
- Open-ended generation supports zero-shot use: a model receives a prompt containing a test instance and produces an output without parameter updates.The passage gives GPT-3 as an example and notes that many tasks can be framed as open-ended or text-to-text generation.Source: §1 · p. 1
- The paper focuses on pre-existing technical bias: systematic discrimination or unequal treatment toward a demographic, originating in human data and entering technical systems.The cited definition of bias is attributed to Sun et al. (2021).Source: §1 · p. 1
- Bias prompts elicit model completions for social groups, which are assessed with metrics such as toxicity or sentiment.The resulting measurements inform debiasing efforts and real-life deployment.Source: §1 · pp. 1–2
- Different prompt sets, metrics, automatic tools, and decoding settings can substantially shift bias results, making measurements vulnerable to incidental design choices.The authors warn that this can lead researchers to miss biases or exaggerate them.Source: §1 · p. 2
2 Background
The background connects open-ended generation to user-facing applications and the need to test for asymmetric behavior across social groups. It describes existing prompt-and-metric comparisons and distinguishes this paper’s focus on how measurement choices shift conclusions.
2 Background
- Open-ended generation can be configured with left-to-right, encoder-decoder, or encoder-only models, among other architectures.The passage gives n-grams, recurrent neural networks, Transformer decoders, and BERT as examples.Source: §2 · p. 2
- Open-ended question answering and dialogue are user-facing applications, and asymmetric hurtful outputs across groups motivate proactive bias testing.The passage notes that users’ prompts may be beyond system owners’ control.Source: §2 · p. 2
- Prior work uses prompts across race, gender, and religion, then compares group-conditioned completions with metrics such as sentiment and toxicity (Sheng et al., 2019; Dhamala et al., 2021).Gender prompts commonly use binary-gendered subjects or strongly gender-associated names.Source: §2 · p. 2
- The paper focuses on open-ended text generation and asks how experimental-setting choices can shift bias conclusions, rather than claiming bias in a particular model.It situates this focus alongside prior critiques of NLP bias quantification (Blodgett et al., 2020; Dev et al., 2021).Source: §2 · p. 2
3 Method
The method tests whether bias results change under alternative experimental settings. It compares metric ratios across groups using generations prompted by contrasting sequences and varies measurement tools, decoding, sample size, and prompt sets.
3 Method
- The measurement scheme uses bias-prompt benchmarks made of contrasting sequence pairs, following prior work (Sheng et al., 2019; Dhamala et al., 2021).The passage names GPT-2 and GPT-3 as models used in the study, though its model-description text is unclear.Source: §3 · p. 2
- The analysis varies metrics, automatic measurement tools, decoding parameters, sample sizes, and bias prompt sets to test result brittleness.Metrics are computed over the entirety of each prompt and completion.Source: §3 · p. 3
- The ratio compares metric values for generations from groups G1 and G2, with φ_h denoting an automatic tool for metric h.The authors use the ratio for brevity and compute metrics on prompts plus completions because partial completions may be out-of-domain for metric tools.Source: §3 · p. 3
4 Findings
Across BOLD and Nozza et al. prompts, the findings examine how metrics, automatic tools, decoding settings, sample choices, and prompt sets affect bias measurements. Experiments generally use 20 completions per prompt and aggregate results unless noted; some toxic-labeled completions do not refer to the protected identity.
4.1 Alternative Metrics to Compare
- Female prompts are 1.4 times more toxic than males at T = 50, versus a toxicity ratio of 1.07 at T = 10; sentiment is insensitive to T, and metrics yield different bias conclusions.The figure compares toxicity, sentiment, and regard ratios for gender prompts in BOLD using GPT-2 generations.Source: Figure 2 · p. 3, §4.1 · p. 3
4.2 Automatic Tools for Metrics
- VADER gives a female/male sentiment ratio of 1.03, while DistilBERT gives 0.94, pointing in opposite directions despite both being close to 1.The comparison uses default GPT-2 decoding parameters on BOLD.Source: §4.2 · pp. 3–4
4.3 Decoding Parameters
- Sikhism/Hinduism toxicity ratios are 18 at T = 10 and 3.8 at T = 40; temperature can also flip the gender-comparison direction.The reported ratios vary non-monotonically with token count; at T = 10, completions are more toxic for males at τ = 0.6 than at τ = 1.0.Source: §4.3 · p. 4
4.4 Sample Size
- For Hispanic/Latino prompts, the toxicity ratio falls from 1.32 for top-1 to 1.05 for top-10; gender ratios also decrease from 1.14 at top-1 to 1.03 at top-5.The race comparison is against European Americans; the gender comparison uses T = 10.Source: Figure 5 · p. 5, §4.4 · p. 5
4.5 Prompting Set Choice
- At T = 10, BOLD and Nozza et al. toxicity ratios are 1.27 and 1.07, respectively; their changes across token counts also differ.From T = 40 to T = 50, BOLD shows no change while Nozza et al. shows a 6% decrease.Source: Table 1 · p. 5, §4.5 · p. 5
4.6 Ambiguous Prompts
- Some HONEST completions labeled toxic do not refer to the prompt’s protected identity; the authors recommend situation-first prompts that examine gendered words in completions.One suggested prompt begins, “Math is difficult for”.Source: Table 2 · p. 5, §4.6 · p. 5
5 Recommendations for Future Work
The paper recommends reporting bias results in a bidimensional scheme that makes variation across metrics, tools, prompt sets, and decoding settings visible. It also recommends human evaluation of a sample of generations and adapting the table to the study’s contribution.
5 Recommendations for Future Work
- Report results across metrics and tools in columns and across prompt sets and decoding settings in rows, using a bidimensional leaderboard.The table can add rows for a new prompt set or columns for a new metric.Source: §5 · pp. 5–6, Table 3 · p. 6
- Include human evaluation on a subsample of generations because automatic tools may partly fail to capture human judgments.The reporting table’s example uses five GPT-3 completions per prompt and average scores.Source: §5 · p. 6, Table 3 · p. 6
- For a debiasing technique, repeat the full table or report percent changes in parentheses to show posterior effects after intervention.Researchers should select a plausible, representative subset of decoding schemes for their applications.Source: §5 · p. 6, Table 3 · p. 6
6 Bias Statement and Limitations
The authors warn that experimental choices can produce inconsistent bias readings, complicating efforts to identify representational harm and alleviate bias. They also identify limitations in their own analysis, including its simple ratio metric and reliance on selectively chosen benchmark prompts and demographics.
6 Bias Statement and Limitations
- Inconsistent results and disagreement over their interpretation can make bias in NLP systems seem subjective rather than a matter for shared understanding.The authors say these inconsistencies can also confuse bias alleviation and hinder identification of representational harm.Source: §6 · p. 6
- Ad-hoc settings can produce dramatically different bias results, and differences may let ill-intentioned actors skew analyses to obscure bias against a demographic.The authors hypothesize that underuse of domain-expert annotators and underprioritization of settings contribute to inconsistent measurements.Source: §6 · p. 6
- The analysis uses a simple ratio-based metric, while its resources are constrained by benchmark choices that selectively cover prompts and demographics.The authors note that more intricate metrics may be considered.Source: §6 · p. 6
7 Conclusion
Across gender, race, and religion, experimental design choices significantly affect measured bias results. The authors emphasize comprehensive reporting to help address technical design biases and misrepresentation of bias outlooks.
7 Conclusion
- Experimental settings and design choices significantly affect bias results across gender, race, and religion.The choices include prompt sets, metrics, and automatic tools used to measure metrics.Source: §7 · p. 7
- The conclusion frames this evaluation around open-ended text generation models.Source: §7 · p. 7
- The authors call for more comprehensive reporting to alleviate technical design biases and misrepresentation of bias outlooks.Source: §7 · p. 7
A Religious Identity Experiments for GPT-2
The GPT-2 experiments compare religious-group toxicity with Christianity across different numbers of generated tokens. Figure 6 defines the comparison as a ratio of mean toxicity scores.
A Religious Identity Experiments for GPT-2
- The experiments compare generations’ toxicity for Christianity with other religions across different numbers of generated tokens.Source: A Religious Identity Experiments for… · p. 9
- Figure 6 divides each religion’s mean generation-toxicity score by the mean for Christianity.Source: Figure 6 · p. 9
- Figure 6 uses 20 generated samples per prompt, and a ratio above 1 indicates toxicity may be higher for the given religion than for Christianity.Source: Figure 6 · p. 9
B Effect of Sample Size for Gender
For gender, which samples are included in aggregate statistics have a non-negligible effect on the magnitude and direction of measured bias.
B Effect of Sample Size for Gender
- The samples included in aggregate statistics have a non-negligible effect on bias magnitude.Source: B Effect of Sample Size for Gender · p. 9
- The samples included in aggregate statistics also have a non-negligible effect on bias direction.Source: B Effect of Sample Size for Gender · p. 9
- The reported effect concerns which samples are considered when computing aggregate statistics for language generation.Source: B Effect of Sample Size for Gender · p. 9
C GPT-3 Experiments
The GPT-3 experiments reaffirm that experimental settings and design choices can have dramatic effects on bias results. The authors also report that temperature settings might increase the female-to-male toxicity ratio measured with detoxify.
C GPT-3 Experiments
- The section reports that additional experiments were conducted using GPT-3.Source: C GPT-3 Experiments · p. 9
- Temperature settings might increase the female-to-male toxicity ratio measured with detoxify.Figure 8 demonstrates this comparison.Source: C GPT-3 Experiments · p. 9
- The GPT-3 experiments reaffirm that benchmark, tool, and metric selection can have dramatic effects on bias results.The authors state that these experiments reaffirm their conclusions about experimental settings and design choices.Source: C GPT-3 Experiments · p. 9
D Top-k for Beam Search Decoding
Top-k beam-search decoding strongly affects toxicity ratios and whether generations are more toxic for one group than another. The female/male ratios range from 0.82 to 1.65 depending on temperature and top-k.
D Top-k for Beam Search Decoding
- 0.82 to 1.65: female/male toxicity ratios vary with the combination of temperature and top-k.The paper reports that top-k strongly affects both toxicity and which group’s generations are more toxic.Source: D Top-k for Beam Search Decoding · p. 9
- Top-k strongly affects toxicity and whether generations are more toxic for female or male prompts.Source: D Top-k for Beam Search Decoding · p. 9
- The comparison uses GPT-2 prompted with Nozza et al. (2021), reporting female/male toxicity ratios across beam-search top-k values; top-k=1 is greedy decoding.Source: Figure 9 · p. 9