Source-linked AI summary
On Measuring Social Biases in Sentence Encoders
Chandler May, Alex Wang, Shikha Bordia, Samuel R. Bowman, Rachel Rudinger
TL;DR
Existing word-level bias tests provide limited evidence about sentence encoders and cannot readily capture some contextual stereotypes. This paper extends WEAT into SEAT and evaluates sentence encoders on established and sentence-level social-bias tests, finding mixed evidence with sensitivity and validity concerns.
Problem
Word-level bias findings do not establish whether sentence encoders are similarly biased, and some important social biases are difficult or impossible to test at the word level.
Method
The paper extends WEAT into SEAT by applying association testing to sentence vectors, including templated sentences and context-specific tests.
Results
Sentence encoders show varying evidence of human-like bias: sentence-to-vector models largely exhibit tested stereotypes, whereas BERT shows limited evidence, with results varying across tests and models.
Takeaways & Limitations
SEAT detects bias in sentence encoders but its results contain idiosyncrasies suggesting limited external validity and a need to examine bias-measurement methods critically.
Takeaways & Limitations
SEAT has positive predictive ability only, so a lack of significant associations does not establish that a model is bias-free.
Abstract
from arXiv · showhide
The Word Embedding Association Test shows that GloVe and word2vec word embeddings exhibit human-like implicit biases based on gender, race, and other social constructs (Caliskan et al., 2017). Meanwhile, research on learning reusable text representations has begun to explore sentence-level texts, with some sentence encoders seeing enthusiastic adoption. Accordingly, we extend the Word Embedding Association Test to measure bias in sentence encoders. We then test several sentence encoders, including state-of-the-art methods such as ELMo and BERT, for the social biases studied in prior work and two important biases that are difficult or impossible to test at the word level. We observe mixed results including suspicious patterns of sensitivity that suggest the test's assumptions may not hold in general. We conclude by proposing directions for future work on measuring bias in sentence encoders.
1 Introduction
The paper asks whether sentence encoders inherit the systematic social biases previously found in word embeddings. It extends bias testing to sentence-level representations and introduces tests for biases difficult to study with individual words.
- Sentence encoders are investigated because word embeddings such as word2vec and GloVe encode systematic biases against women and black people.
- The authors generalize WEAT from word sets to phrases and sentences as the Sentence Encoder Association Test (SEAT).
- SEAT adds tests for the angry black woman stereotype and a professional double bind affecting women.
- Sentence-level contexts also enable comparisons between given names and group terms, an experimental variation generally unavailable to WEAT.
- Sentence encoders show varying evidence of human-like bias, while BERT displays limited evidence in the tested biases.
2 Methods
The methods extend WEAT from word embeddings to fixed-sized sentence representations, using sentence templates and statistical association tests to measure bias.
- WEAT compares associations between two equal-sized target-concept sets and two attribute sets using differences in cosine similarities.
- A permutation test estimates the significance of the association by comparing the observed statistic with statistics from equal-sized partitions.
- Effect size is a normalized difference of means, and larger effect sizes indicate more severe bias when significance is controlled.
- SEAT applies WEAT to sentence vectors, pooling variable-length encoder outputs when necessary to obtain fixed-sized representations.
- Words are inserted into semantically bleached templates designed to convey little meaning beyond the inserted terms.
3 Biases Tested
The paper tests previously studied implicit associations alongside sentence-level stereotypes involving race, gender, occupational competence, and interpersonal evaluations.
- Caliskan Tests: Caliskan tests are applied in both original word-level form and generated sentence-level form to assess whether sentence encoders reproduce established human-like biases.
- Angry Black Woman Stereotype: The angry black woman test uses black- and white-identifying female names with stereotype-related adjectives, group terms, and sentence templates.
- Double Binds: The double-bind tests examine women and men represented as engineers, pairing professional contexts with likability, hostility, competence, or achievement-oriented attributes.
- Double Binds: Semantically unbleached tests preserve context relevant to the bias, while word-level and bleached sentence-level controls probe whether associations depend on gender independent of context.
4 Experiments and Results
Experiments apply SEAT to seven sentence encoders and compare word-level, bleached sentence-level, and unbleached sentence-level tests. Results vary across models and tests, with multiple-testing-corrected evidence strongest for Caliskan and angry black woman stereotype tests, while several patterns challenge SEAT’s evaluation assumptions.
- Experimental setup: SEAT is applied to seven sentence encoders spanning bag-of-words, sentence-to-vector, and state-of-the-art sequence models.The models use publicly available pretrained parameters.
- Experimental setup: Given-name tests more often find significant associations than group-term tests.The reported results show only the given-name versions for this reason.
- Bias results: Bleached sentence-level tests tend to yield more significant associations, whereas word-level tests tend to produce larger effect sizes.After correction, evidence is stronger for Caliskan and angry black woman stereotype tests than for the double bind.
- Bias results: After multiple-testing correction, double-bind evidence appears only in bleached sentence-level competent-control tests, associating women with incompetence independent of context.Results differ across models: ELMo and GPT show no significant associations, while USE and BERT show one each.
- Sensitivity patterns: 10^-5 and 0.14 are discrepant sentence-version p-values for BERT on the two related math/art and science/art tests.GenSen and GPT also show discrepant pairs: 0.12 and 10^-3, and 0.89 and 10^-4, respectively.
- Sensitivity patterns: ELMo’s p-values decrease across Caliskan Tests 3–5, reaching 10^-4 on their sentence versions, contrary to the expected increasing pattern.The authors interpret this as substantially different representations for conceptually similar words, undermining SEAT’s coherent-concept assumption.
5 Conclusion
The study extends an established word-level bias instrument to sentence encoders and finds significant effects alongside patterns suggesting limited external validity. It cautions that tested associations do not establish bias absence or generalize beyond the specific inputs and design choices.
- SEAT can detect bias presence, but a negative result does not show that a sentence encoder is bias-free.
- The authors caution that tested biases may not generalize beyond the specific words and sentences used in the tests.
- Possible sensitivity sources include non-bleached templates, aggregation choices, and cosine similarity, motivating future application-level bias measurement.
- The discussion highlights intersectionality, including race-and-gender bias, as a consideration for future fairness and ethics research.
- The authors develop a sentence-level extension of an established word-embedding bias instrument to measure social biases in pretrained sentence encoders.
- The study observes many significant effects, but idiosyncratic sensitivities suggest that the findings have limited external validity.
A Computation of P-value and Effect Size
The computation uses permutation-based significance testing for WEAT statistics and computes effect size as in prior work. It switches to a nonparametric implementation when normality is not satisfied and imposes a sampling resolution limit.
- The p-value is computed from the probability that a statistic from an equal-size partition exceeds the observed statistic.
- The parametric implementation samples 100 000 statistics from the null distribution, fits a normal distribution, and evaluates its upper-tail probability.
- The authors use a nonparametric test when normality is not satisfied, enumerating at most 100 000 partitions or sampling 99 999 partitions otherwise.
- When partitions are sampled, the procedure cannot produce a p-value below 10^-5, or 1/100 000.
- The effect size is computed identically to the prior WEAT implementation using an unbiased estimate of the standard deviation.
B Test Details and Variations
The supplementary test data are distributed as JSON Lines files in the tests directory. The materials include descriptions of the data and test variations used in the study.
- The supplementary test data are provided in JSON Lines files with the .jsonl extension.
- The files are located in the tests directory of the supplementary data.
- The following sections describe the test data and variations on the tests presented in the paper.
B.1 Caliskan Tests
The Caliskan tests evaluate whether sentence encoders reproduce biases previously observed in word embeddings. Supplementary materials distinguish word- and sentence-level test names and document alternate versions that swap names with group terms.
- The Caliskan tests examine whether sentence encoders reproduce biases reported for word embedding models.
- Alternate versions of Tests 3, 5, 6, 7, and 8 swap given names with group terms and use the suffix b.
B.1.1 Example: Caliskan Test 3
The sentence-level Caliskan Test 3 represents racialized names and pleasantness through semantically bleached sentence templates. The surrounding examples also show how sentence-level tests extend to stereotype and double-bind designs with varied contextual detail.
- Caliskan Test 3: Caliskan Test 3 compares European-American and African-American names as targets against pleasant and unpleasant attributes in sentence templates.Examples include “This is Adam” versus “This is Alonzo,” and “This is a caress” versus “This is an abuse.”
- Test variants: The sentence-level tests use given-name and group-term variants, with the sent- prefix marking sentence versions and suffix b marking group-term alternatives.These naming conventions distinguish experimental variants in the supplementary data.
- Angry black woman stereotype: Angry black woman stereotype tests use racialized female names or group terms as targets and stereotype-related adjectives and antonyms as attributes.The sentence version inserts these target and attribute terms into templates.
- Double binds: Double-bind tests vary sentence context and distinguish clear-success likability from ambiguous-success competence conditions.The designs range from short templates to multiple-sentence descriptions of a male-typed aircraft-company leadership role and performance review.
- Double binds: The unbleached competent double-bind example compares male and female engineers using sentences such as “John is an engineer” and “Amy is an engineer.”The supplied examples also identify male and female targets for the likable version and describe the fuller target-concept format.
B.3.4 Example: Likable, Multiple Sentence
The likable multiple-sentence double-bind test evaluates gendered judgments in a richly contextualized workplace scenario. It uses a successful female assistant vice president alongside likability attributes and several sentence-encoder implementations.
- Likable, Multiple Sentence: The fullest likable test describes a female assistant vice president who has excelled in an annual performance review and is a top-5% performer.Her responsibilities include supervising executives, entering markets, tracking industry trends, and generating clients.
- Likable, Multiple Sentence: The likable test pairs the successful workplace target concept with attributes such as agreeable, fair, and honest.The supplied example identifies “The assistant vice president is agreeable” as one attribute element.
- Attribute construction: The paper also constructs antonym-based attribute sets for the angry black woman stereotype and double-bind tests with ad-hoc thesaurus assistance.These antonym sets form part of the attribute design across the relevant tests.
- Sentence encoders: The evaluated sentence encoders range from averaged-word and recurrent models to ELMo, GPT, and BERT Transformers.The implementations differ in architecture and sentence-representation extraction, including pooling, final hidden states, and BERT’s [CLS] state.
- Sentence encoders: BERT represents each sequence with the top hidden state for its [CLS] token, using the 1024-dimensional large cased version in the reported results.GPT instead uses the top hidden state corresponding to the last word.
D Results
The results section provides complete statistical outputs and a visualization of significant model-by-test results. The supplementary results file records model, test, significance, effect-size, and set-size information.
- Results data: The supplementary results.tsv file contains the complete set of model-by-test results.It is a tab-separated file whose first row names the columns.
- Multiple testing correction: Holm-Bonferroni correction is computed across all non-header rows in results.tsv at α = 0.01 after sorting hypotheses by increasing p-value.The procedure uses the rank of each p-value to determine corrected significance.
- Results visualization: Figure 1 visualizes significant results across the matrix of models and bias tests.Its stated purpose is to summarize which model-test combinations are significant.
- Results data: The results file records model name, model options, test name, p value, effect size, and the sizes of two target and two attribute sets.These columns identify both the tested configuration and the corresponding statistical quantities.
- Results data: Table 5 is the captioned reference for the names and descriptions of the results.tsv columns.The supplied materials include both the table caption and a duplicate figure-style caption entry.