Source-linked AI summary
Social Biases in NLP Models as Barriers for Persons with Disabilities
Ben Hutchinson, Vinodkumar Prabhakaran, Emily Denton, Kellie Webster, Yu Zhong, Stephen Denuyl
TL;DR
The paper asks whether NLP models encode social biases toward disability, an important question because such models increasingly mediate interactions with technology. It evaluates classification models, language representations, and disability-related discourse, finding measurable biases across these components and identifying over-represented social associations in the data. The authors conclude that understanding model bias requires attention to socio-technical factors and stakeholder involvement.
Problem
The paper addresses limited evidence about how NLP models represent and perpetuate social biases toward persons with disabilities, a concern for increasingly deployed technologies.
Method
The authors perturb disability references in text-classification and BERT analyses, then examine disability-related terms and categories in a public corpus.
Results
The study finds measurable disability biases in classification models and language representations, while corpus analysis identifies over-represented social associations in disability-related comments.
Takeaways & Limitations
The findings support considering socio-technical factors, training data, and known model biases when assessing NLP systems involving disability.
Takeaways & Limitations
The study is limited to English and US sociolinguistic norms, so whether its findings carry over to other languages and cultural contexts remains open.
Abstract
from arXiv · showhide
Building equitable and inclusive NLP technologies demands consideration of whether and how social attitudes are represented in ML models. In particular, representations encoded in models often inadvertently perpetuate undesirable social biases from the data on which they are trained. In this paper, we present evidence of such undesirable biases towards mentions of disability in two different English language models: toxicity prediction and sentiment analysis. Next, we demonstrate that the neural embeddings that are the critical first step in most NLP pipelines similarly contain undesirable biases towards mentions of disability. We end by highlighting topical biases in the discourse about disability which may contribute to the observed model biases; for instance, gun violence, homelessness, and drug addiction are over-represented in texts discussing mental illness.
1 Introduction
The paper examines disability-related social biases in NLP models, addressing an under-explored issue affecting a large population. It evaluates classification and language models and analyzes data patterns that may help explain observed biases.
- The paper studies whether NLP models classify or predict disability-related text in ways that can affect people writing, reading, or seeking disability information.
- Disability-related bias in NLP is under-explored despite affecting over one billion people, about 15% of the world’s population.
- The authors demonstrate measurable disability biases in two text-classification models, with bias strength depending on how disability is mentioned.
- They also find measurable disability biases in language models used as inputs to downstream NLP systems.
- A corpus analysis identifies social biases in data that provide a likely explanation for the observed model biases.
2 Linguistic Phrases for Disabilities
The study represents disability references with a curated set of English expressions classified by prescriptive language guidelines, while recognizing that this binary classification is coarse.
- The analysis uses 56 English expressions for referring to people with various disabilities, divided into Recommended and Non-Recommended groups.
- The recommended-status partition draws on guidelines from the Anti-Defamation League, ACM SIGACCESS, and the ADA National Network.
- The authors acknowledge that the binary distinction is only a coarse-grained view of disability language.
3 Biases in Text Classification Models
The paper measures how replacing pronouns with disability references changes toxicity and sentiment model scores, finding elevated toxicity associations that can penalize innocuous disability-related text.
- The perturbation method replaces he or she in naturally occurring sentences with disability phrases and computes the resulting model score difference.
- 0.007 versus 0.057: recommended phrases had a smaller aggregate toxicity score difference than non-recommended phrases.
- All disability categories showed varying degrees of association with toxicity, including stronger effects for some recommended phrases.
- The toxicity model’s moderation use could cause non-toxic comments mentioning disabilities to be flagged disproportionately and innocuous discussions to be suppressed.
- The sentiment model showed similar patterns involving both desirable and undesirable associations when disability phrases replaced pronouns.
4 Biases in Language Representations
The paper finds that language representations associate disability references with negative sentiment and examines how such associations may enter downstream NLP applications. It also describes a BERT fill-in-the-blank analysis of these effects.
- Neural text embeddings encode semantic relationships geometrically but may also encode undesirable correlations reflecting harmful social biases.
- BERT is analyzed by perturbing templates containing disability phrases and observing changes in its top-ranked fill-in-the-blank predictions.
- The sentiment of BERT-predicted words is evaluated in neutral sentences to isolate differences among predicted words rather than disability-phrase sentiment itself.
- For most disability phrases, a larger percentage of BERT’s predicted words received negative sentiment scores, suggesting negative associations in its representations.
5 Biases in Data
The paper examines which terms co-occur with disability mentions in online comments and finds that social associations, including negatively valenced topics, are prominent. These corpus patterns may contribute to how disability terms are represented in NLP models.
- Corpus analysis: The analysis uses a balanced 15,436-comment dataset to identify terms over-represented in comments mentioning psychiatric or mental illness.The dataset contains four equally sized groups formed by balancing disability mentions and toxic versus non-toxic comments.
- Corpus analysis: Over-represented terms are grouped into five categories: CONDITION, TREATMENT, INFRASTRUCTURE, LINGUISTIC, and SOCIAL.The categories distinguish disability descriptions, care, support infrastructure, linguistic associations, and social associations.
- Corpus analysis: Table 3 reports the ten most associated terms in each category using log-odds ratio scores.The scores quantify the strength of each term’s association with disability mentions.
- Findings: CONDITION phrases have the highest association, while SOCIAL phrases have the next highest association, exceeding TREATMENT, INFRASTRUCTURE, and LINGUISTIC phrases.This comparison concerns the category-level associations reported in the corpus analysis.
- Findings: The SOCIAL category largely concerns homelessness, gun violence, and drug addiction, topics with negative valences often discussed alongside mental illness.The paper notes that these associations significantly shape disability-term representations and may contribute to previously observed model biases.
6 Implications of Model Biases
Model biases can affect how people participate in online forums and how their messages are moderated, reviewed, or challenged. The paper argues that deployment context and socio-technical considerations are essential for interpreting these effects.
- Socio-technical considerations: Technical fairness analysis should be supplemented by normative, social, and stakeholder considerations when NLP systems are applied.The paper emphasizes impacts on people affected directly or indirectly, voluntarily or involuntarily, by deployment.
- Online moderation: Biases in abuse-detection models may disproportionately censor messages mentioning disability, particularly without human moderation.This could affect equal participation in online forums and readers’ exposure to disability-related discussion.
- Online moderation: Model-score prioritization can determine how long different authors’ messages wait for moderator approval.The practical effect depends on whether higher- or lower-scoring messages are reviewed first.
- Writer nudging: Abuse-detection systems that nudge writers may disproportionately invalidate language choices when people write about disabilities.The paper identifies potential disrespect and offense as consequences in this deployment context.
- Deployment boundaries: Deployment data distributions that differ from development data may reduce model robustness and exacerbate these issues.The paper recommends documenting intended and unintended uses, training data, and known biases when releasing models.
7 Discussion and Conclusion
The paper concludes that disability-related social biases occur across NLP models and can shape engagement with technology and negative stereotypes. It also highlights boundaries imposed by English-language and US sociolinguistic scope.
- Discussion and conclusion: Evidence across three readily available NLP models shows that disability-related social biases warrant concern because models can moderate technology engagement and perpetuate negative stereotypes.The paper reports sensitivity to disability types and to the prescriptive status of referring expressions.
- Limitations and future work: The study is limited to English and US sociolinguistic norms, so future work must examine other languages and cultural contexts.The paper also notes that disability phrases and ontological definitions are contested across people and groups.
- Limitations and future work: Defining mitigation objectives for ableism in ML should involve disability communities and other impacted stakeholders.This recommendation follows the paper’s observation that language and disability categories are contested.
A.1 Expressions for Disability
The paper’s expression set uses guideline-based categories to distinguish recommended from non-recommended ways of referring to people with disabilities. The authors also classify the phrases into author-defined categories.
- Expression sets: The experiments use recommended and non-recommended phrases drawn from guidelines published by the ADL, SIGACCESS, and ADA National Network.Table 6 presents recommended phrases, while Table 7 presents non-recommended phrases.
- Expression sets: The authors, rather than the guideline organizations, grouped the expressions into categories.The passage explicitly attributes the category grouping to the authors.
A.2 Tabular versions of results
The tabular results provide accessible presentations of the BERT negative-sentiment frequencies, including top-10 word suggestions produced with recommended disability phrases.
- Table 4 reports how often BERT’s top-10 word suggestions produce negative sentiment scores when recommended disability phrases are used.
A.3 Text classification analyses for individual phrases
The analyses test how individual disability phrases affect toxicity and sentiment model scores and examine BERT’s negative-sentiment word predictions. Results show phrase-sensitive model behavior, with some predicted negative words excluded because they also occur without disability phrases.
- Figures 3 and 4 measure toxicity and sentiment model sensitivity to individual disability phrases.
- Seven hand-crafted templates based on gender-neutral references were perturbed by inserting recommended disability phrases.
- BERT’s fill-in-the-blank analysis examines words predicted after sentences containing disability terms and producing negative sentence scores.
- Three negative words—“disqualified,” “excluded,” and “registered”—were omitted because BERT also produced them for sentences without disability phrases.
- Table 5 compares average toxicity or sentiment score changes for recommended and non-recommended disability phrases against pronouns.
- Table 6 lists recommended phrases used to analyze model sensitivity, while Table 7 documents the corresponding non-recommended phrases.