PaperlayerView exact revision on arXiv

Source-linked AI summary

BBQ: A Hand-Built Bias Benchmark for Question Answering

Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, Samuel R. Bowman

arXiv:2110.08193v2cs.CL

TL;DR

BBQ addresses the limited evidence on how social biases manifest in QA model outputs. It introduces a hand-built benchmark of attested biases and tests models with both under-informative and disambiguated contexts. Models often rely on stereotypes when information is insufficient, and bias remains associated with accuracy differences even when the correct answer is provided.

  • Problem

    Little work has examined how social biases manifest in question-answering model outputs, despite evidence that language models reproduce biases in other tasks.

  • Method

    BBQ uses hand-written, validated question sets targeting attested biases across nine categories, testing ambiguous contexts and contexts that explicitly provide the correct answer.

  • Results

    Models generally select unsupported, bias-aligned answers in under-informative contexts, while disambiguated contexts show up to 3.4 percentage points lower accuracy when the correct answer conflicts with bias.

  • Takeaways & Limitations

    BBQ measures when QA model behavior may reinforce stereotypes or override correct answers, supporting analysis of which biases require mitigation or further study.

  • Takeaways & Limitations

    BBQ is scoped to short contexts and biases relevant to nine categories in U.S. English-speaking cultural contexts, so it may not validly measure behavior in other domains or contexts.

Abstract

from arXiv · show

It is well documented that NLP models learn social biases, but little work has been done on how these biases manifest in model outputs for applied tasks like question answering (QA). We introduce the Bias Benchmark for QA (BBQ), a dataset of question sets constructed by the authors that highlight attested social biases against people belonging to protected classes along nine social dimensions relevant for U.S. English-speaking contexts. Our task evaluates model responses at two levels: (i) given an under-informative context, we test how strongly responses reflect social biases, and (ii) given an adequately informative context, we test whether the model's biases override a correct answer choice. We find that models often rely on stereotypes when the context is under-informative, meaning the model's outputs consistently reproduce harmful biases in this setting. Though models are more accurate when the context provides an informative answer, they still rely on stereotypes and average up to 3.4 percentage points higher accuracy when the correct answer aligns with a social bias than when it conflicts, with this difference widening to over 5 points on examples targeting gender for most models tested.

1 Introduction

BBQ addresses limited understanding of how social biases appear in QA outputs by introducing a hand-built benchmark targeting attested biases across socially relevant categories. It tests both stereotype reliance under insufficient information and whether stereotypes override explicitly correct answers.

  • Motivation: BBQ targets attested social biases across nine socially relevant categories and covers harms relevant to U.S. contexts.The benchmark uses hand-written contexts validated by experts and crowdworkers.
  • Evaluation: The benchmark tests model responses with insufficient context and with disambiguating information that supplies the correct answer.The first setting measures reliance on uncertainty versus unsupported answers; the second tests whether bias overrides a valid answer.
  • Contribution: BBQ is intended as a stronger measurement tool for drawing reliable conclusions about how models reproduce social biases.The paper does not directly provide debiasing or harm-reduction measures.
  • Findings: Models generally choose unsupported, often bias-aligned answers in under-informative contexts rather than expressing uncertainty.The evaluated models were UnifiedQA, RoBERTa, and DeBERTaV3.
  • Findings: 3.4 percentage points is the maximum accuracy decrease in disambiguated contexts when the correct answer conflicts with a social bias.This shows that bias can persist even when the context provides the needed answer.

2 Related Work

Prior NLP bias work established that models encode social stereotypes and can produce biased outputs in downstream tasks. For QA specifically, BBQ extends earlier likelihood-based evaluation by measuring discrete model predictions and their relationship to correct answers.

  • Measuring Bias in NLP: Existing studies consistently find that NLP models encode associations derived from negative stereotypes and social biases.Definitions of bias vary across NLP studies.
  • Downstream NLP Tasks: Downstream-task research examines how encoded biases affect outputs in coreference resolution and hate speech detection.These tasks provide precedents for studying bias beyond model representations.
  • Coreference Resolution: Coreference studies report gender and gender-occupation biases, including incorrect masculine-pronoun selection and harmful predictions involving non-binary identities.The cited work also connects performance differences to stereotype alignment and annotation effects.
  • Hate Speech Detection: HATECHECK and related studies identify performance differences across target groups, languages or varieties, and demographic categories in hate-speech detection.The literature includes gender, race, age, religion, sexual orientation, and disability status.
  • Question Answering: UnQover evaluates QA bias through likelihoods for answer options in underspecified questions involving gender, occupation, nationality, ethnicity, and religion.It does not directly measure whether stereotypes alter categorical model outputs.

3 The Dataset

BBQ constructs controlled question sets from hand-written templates that isolate attested social biases across nine categories and two intersectional categories. Each set contrasts ambiguous and disambiguated contexts and negative and non-negative questions to evaluate both uncertainty and bias-specific responses.

  • Dataset Construction: BBQ contains 58,492 unique examples built from templates with partial contexts, disambiguating contexts, questions, and answer choices.Templates isolate a target bias while varying category labels in consistent contexts.
  • Evaluation Design: BBQ compares model predictions directly and always includes a correct answer option plus UNKNOWN alternatives.This distinguishes output-changing bias from likelihood differences and tests when bias overrides a valid answer.
  • Bias Categories: The dataset covers nine broad social-bias categories and two intersectional categories, including gender-by-race/ethnicity and socioeconomic-status-by-race/ethnicity.Intersectional examples are analyzed separately because their comparisons cannot be randomized in the same way.
  • Template Construction: Each template is hand-written around a documented harmful bias and supported by evidence such as research, news, Wikipedia, or first-hand accounts.Intersectional templates compare entities differing along one or both targeted dimensions.
  • Question Design: Each example cluster combines negative and non-negative questions with and without disambiguating context.The paired question types help separate question-specific biased associations from general preferences for a label.
  • Context Design: Ambiguous contexts have UNKNOWN as the correct answer, whereas disambiguated contexts identify one of the two individuals as correct.The two context types isolate insufficient-evidence behavior from failures to follow explicit information.
  • Template Coverage: The nine top-level categories and two intersectional categories each include 25 unique templates, with additional proper-name templates for race and gender.Most templates generate at least 100 examples, though the range is 8 to 200.
  • Label Design: Proper names are treated as imperfect proxies for race and gender, so name-based templates are analyzed separately from explicit category-label templates.The vocabulary reflects extreme distributional differences in name-category associations.

4 Validation

BBQ examples were validated through repeated crowdworker judgments, with templates edited and revalidated when any condition failed the inclusion threshold. Human performance and agreement were high on a random sample of the final dataset.

  • Validation Procedure: Each template contributed one randomly sampled item from each of its four conditions to a multiple-choice validation task.Five annotators rated each item.
  • Validation Procedure: Templates were included only when at least 4/5 annotators agreed with the gold label across the validation items.Templates below threshold were edited and all four associated items were revalidated.
  • Human Performance: 95.7% is the estimated raw crowdworker accuracy on BBQ, compared with 99.7% aggregate accuracy by majority vote.These estimates come from a random sample of 300 final-dataset examples.
  • Human Agreement: Krippendorff’s α is 0.883, indicating high agreement between raters.

5 Evaluation

BBQ evaluates QA models with separate accuracy and bias measures across ambiguous and disambiguated contexts. The evaluation distinguishes whether models select UNKNOWN, reproduce social biases, and favor bias-aligned correct answers.

  • Models: BBQ evaluates UnifiedQA, RoBERTa, and DeBERTaV3 models using exact-match answer scoring, including multiple input formats and model sizes.UnifiedQA is tested with RACE-style and ARC-style formats; RoBERTa and DeBERTaV3 are fine-tuned on RACE.
  • Metrics: Accuracy is computed by category and context, with disambiguated examples separated according to whether the correct answer reinforces or conflicts with a social bias.This separation tests whether model performance changes when social bias is useful for answering.
  • Metrics: The bias score measures the percentage of non-UNKNOWN outputs aligned with a social bias, separately for ambiguous and disambiguated contexts.Positive scores indicate bias-aligned outputs, while -100% indicates that all answers go against the bias.
  • Metrics: Ambiguous-context bias scores are scaled by accuracy because biased answers are more harmful when they occur more often.This scaling is not applied to disambiguated contexts because their bias score is not computed solely from incorrect answers.
  • Metrics: Accuracy and bias score capture different behaviors because identical accuracy values can arise from different patterns of incorrect answers.Perfect accuracy produces a bias score of zero, but lower accuracy does not uniquely determine the distribution of biased errors.

6 Results

Models were substantially more accurate with disambiguating information, yet their errors and accuracy differences still reflected social biases. Bias reliance was strongest in ambiguous contexts and varied across categories, labels, and stereotypes.

  • Accuracy: 77.8% was the highest overall BBQ accuracy, achieved by UnifiedQA with RACE-style input, while RoBERTa-Base was lowest at 61.4%; chance was 33.3%.Models were generally more accurate in disambiguated contexts than ambiguous contexts.
  • Accuracy: Accuracy remained higher when the correct disambiguated answer aligned with a social bias than when it conflicted with that bias.This difference appeared in most categories within each model.
  • Bias scores: Bias scores were much stronger in ambiguous than disambiguated contexts, partly because higher disambiguated accuracy moves scores closer to zero.In ambiguous contexts, models often failed to select UNKNOWN and instead relied on social biases.
  • Category differences: Physical-appearance biases drove ambiguous-context responses more than race or sexual-orientation biases across the tested models.Gender effects varied by model and were stronger when choosing between gendered names than identity labels for some larger models.
  • Bias scores: About 77% of UnifiedQA’s ambiguous-context errors aligned with the targeted social bias, and this pattern became more pronounced for models stronger on typical NLP benchmarks.Every model produced bias-aligned answers in more than half of its ambiguous-context errors.
  • Category differences: Race/ethnicity results for UnifiedQA showed strong bias against labels such as “Black” and “African American,” with variation across targeted stereotypes.Associations involving anger and violence differed across labels, likely reflecting training-data membership and contextual differences.
  • Intersectional biases: Intersectional-template results were less consistent than non-intersectional results, so the authors could not conclude that models respond sensitively to multiple identity dimensions.The authors attribute the reduced consistency possibly to greater overlap between answer options.
  • Baselines: UnifiedQA’s question-only baseline had accuracy and bias scores that did not substantially differ from its ambiguous-context results.Removing target-entity introductions did not clearly increase UNKNOWN selection or reliance on biases.

7 Discussion

The authors caution that BBQ bias scores are limited in interpretation and cultural scope, while detailed examples show how specific stereotypes can persist even with disambiguating information.

  • Interpretation of Bias Scores: Bias scores near zero do not prove that a model is unbiased because they summarize directionally consistent behavior over only 25 templates per category.A near-zero aggregate can reflect balanced responses to negative and non-negative questions rather than absence of bias.
  • Interpretation of Bias Scores: BBQ is designed for biases associated with US English-speaking cultural contexts and should not be treated as evidence about bias in other cultural settings.The authors also warn that individual template results represent small samples and should be interpreted cautiously.
  • Categories with High Bias Scores: Physical-appearance categories showed high ambiguous-context bias that persisted on some disambiguated templates, indicating occasional replacement of explicit evidence with social stereotypes.Obesity-related examples primarily drove the high ambiguous-context score.
  • Categories with High Bias Scores: A UnifiedQA template associated obese individuals with sloppiness, illustrating potential attribution of a stereotyped property to an individual in QA.The example set contained only 72 examples, making the individual-template result susceptible to noise.

8 Conclusion

BBQ measures how social biases targeting nine categories manifest in QA outputs across underspecified and informative contexts. Models strongly rely on social biases when contexts are underspecified, and biases can still override clear answers.

  • BBQ is a hand-built benchmark measuring social biases targeting nine categories in QA outputs across different context types.
  • Models achieve no more than 67.5% accuracy in ambiguous contexts, with errors reinforcing stereotypes up to 77% of the time.
  • Even when short contexts provide clear answers, social biases occasionally affect accuracy and outputs, causing models to select harmful alternatives.

9 Ethical Considerations

BBQ is intended to measure social biases in QA models, but its conclusions are bounded by its output-based design, included categories, and U.S. English-speaking context. The authors warn that low bias scores should not be interpreted as evidence that a model is unbiased in all cases.

  • A low BBQ bias score should not be taken to indicate that a model is less biased in all cases.
  • BBQ measures behavior in very short contexts for included biases, which are limited to a current U.S. English-speaking cultural context.
  • Because BBQ measures model outputs rather than likelihoods, it may miss biases that UnQover detects, although BBQ more directly tests whether biases manifest in outputs.
  • The benchmark is designed to help researchers identify when and against which groups QA models may reproduce social biases.
  • Vocabulary choices are constrained when categories lack single identity labels, so templates use grammatically fitting, semantically coherent comparisons validated through human inference.

B Proper Name Selection Process

BBQ’s proper-name selection combines demographic association data, frequency thresholds, and category-specific construction rules. The dataset also uses category-specific template counts and human validation procedures to support its examples.

  • Names are included because they convey gender and racial identity information that identity labels alone may not capture.
  • Asian stereotypical names pair gendered Anglophone given names with common Asian-American family names because Asian names are often less strongly gender-associated in U.S. usage.
  • Black, Hispanic/Latinx, White, and Asian name lists use demographic association sources alongside minimum frequency or association thresholds.
  • Middle-Eastern/Arab names were assembled from Wikipedia because the U.S. Census data used for other identities does not provide an applicable category.
  • Each top-level and intersectional category has 25 unique templates, while intersectional categories generate more examples because each template requires three comparisons.
  • Human validation used filler items to prevent annotators from inferring answers from context length, and templates failing agreement thresholds were edited.

E Overall Accuracy Results

Models are substantially more accurate when contexts identify the answer, while ambiguous or question-only settings expose uncertainty-handling and model-specific differences. RoBERTa-Base is an exception, showing higher ambiguous-context accuracy but lower disambiguated-context accuracy because it overselects UNKNOWN.

  • Models are much more accurate in disambiguated contexts than ambiguous contexts, with substantial accuracy variation across categories.
  • RoBERTa-Base has lower overall disambiguated-context accuracy but higher accuracy in most ambiguous-context cases than the other models.This pattern is linked to its greater tendency to answer UNKNOWN regardless of context.
  • The question-only baseline produces results similar to ambiguous contexts in both bias score and accuracy.The baseline removes the context, and UNKNOWN is always correct.
  • Each of the 10 UNKNOWN strings accounts for 4.9%–15.5% of a model’s UNKNOWN outputs, mostly consistent with roughly equal selection rates.Using multiple strings tests whether lexical variation drives model behavior.

H Detailed Results from Intersectional Categories

Intersectional bias effects are generally weak and inconsistent, though race-by-SES patterns and selected race-by-gender targets show suggestive signals. The interpretation of race-by-SES results is constrained by how SES information appears across contexts.

  • Overall intersectional results: Intersectional bias scores are generally close to zero, indicating weaker effects than those observed for non-intersectional categories.
  • Race by SES: Race-by-SES scores are negative across all three comparisons for three higher-performing models, with stronger effects in ambiguous contexts.This pattern indicates greater selection of the bias target for negative questions and the non-target for non-negative questions.
  • Race by gender: UnifiedQA shows systematic race-by-gender intersectional bias for Middle-Eastern women and Asian men, while other labels are more mixed.
  • Race by SES: In ambiguous race-by-SES contexts, positive bias scores most strongly target Hispanic/Latino and Black/African American individuals with lower SES when both dimensions differ.Other identity labels do not consistently follow this pattern.
  • Race by gender: Race-by-gender breakdowns are inconsistent across comparisons, suggesting BBQ does not measure a significant intersectional effect in that analysis.
Loading 2110.08193v2…