Source-linked AI summary
Toward Fairness in AI for People with Disabilities: A Research Roadmap
Anhong Guo, Ece Kamar, Jennifer Wortman Vaughan, Hanna Wallach, Meredith Ringel Morris
TL;DR
Fairness in AI for people with disabilities has received little attention despite AI’s potential to benefit them and the possibility that systems may fail or discriminate. This position paper identifies possible risks across AI and disability categories as a first step toward a broader research roadmap. It concludes that further work is needed on inclusive design guidelines, datasets, algorithms, and error metrics.
Problem
Fairness in AI for people with disabilities has received little attention, despite AI’s potential to help them and the risk that systems may fail or discriminate.
Source: INTRODUCTION · p. 1Method
The paper assesses how classes of AI systems may interact with disability constituencies and proposes a research agenda for addressing inclusion gaps.
Source: INTRODUCTION · p. 1, RISK ASSESSMENT OF EXISTING AI SYSTEMS… · p. 2Results
The authors identify potential ways that AI systems and techniques may limit their efficacy and fairness for people with disabilities.
Source: INTRODUCTION · p. 1, CONCLUSION · p. 5Takeaways & Limitations
The roadmap points toward design guidelines, datasets, algorithmic techniques, and error metrics to help AI benefit people with disabilities while avoiding outlined pitfalls.
Source: CONCLUSION · p. 5The paper describes AI categories and their possible interactions with disability without endorsing whether those systems should be built; that ethical question may depend on context.
Source: INTRODUCTION · p. 2
Abstract
from arXiv · showhide
AI technologies have the potential to dramatically impact the lives of people with disabilities (PWD). Indeed, improving the lives of PWD is a motivator for many state-of-the-art AI systems, such as automated speech recognition tools that can caption videos for people who are deaf and hard of hearing, or language prediction algorithms that can augment communication for people with speech or cognitive disabilities. However, widely deployed AI systems may not work properly for PWD, or worse, may actively discriminate against them. These considerations regarding fairness in AI for PWD have thus far received little attention. In this position paper, we identify potential areas of concern regarding how several AI technology categories may impact particular disability constituencies if care is not taken in their design, development, and testing. We intend for this risk assessment of how various classes of AI might interact with various classes of disability to provide a roadmap for future research that is needed to gather data, test these hypotheses, and build more inclusive algorithms.
INTRODUCTION
Fairness for people with disabilities (PWD) has received little attention despite AI’s potential both to help and to disadvantage them. The paper outlines a research agenda and begins it by identifying possible risks across AI and disability categories.
INTRODUCTION
- Fairness for PWD has received little attention, while systems may exclude them from technologies, amplify stereotypes, or endanger their safety.The paper notes that people with particular disabilities may be overlooked as outliers even when represented in training and evaluation data.Source: INTRODUCTION · p. 1
- The proposed agenda calls for identifying inclusion issues, testing hypotheses and mitigation techniques, creating ethical benchmark datasets, and developing new models, mitigation methods, and error measures.The paper focuses mainly on the first step: identifying ways that inclusion gaps in training and evaluation may affect fairness for PWD.Source: INTRODUCTION · p. 1
- This position paper reflects on how current AI classes may interact with disability classes; systematic study of these interactions remains a next step toward inclusive AI.The authors describe problem articulation as a necessary precursor to remediation.Source: INTRODUCTION · p. 1
- The paper uses “disability” in the social-model sense: impairment results in disability through non-accommodating social or environmental conditions.It also uses people-first language while recognizing that some people prefer identity-first or other terminology.Source: INTRODUCTION · p. 1
- Mentioning AI categories describes possible interactions with disability, not an endorsement of building those systems; whether they are ethical may depend on context.The authors note a potential trade-off: restricting synthetic voices associated with deepfakes could also limit technology that reproduces the voice of someone who can no longer speak.Source: INTRODUCTION · p. 2
RISK ASSESSMENT OF EXISTING AI SYSTEMS FOR PWD
The risk assessment groups AI systems by function and identifies disability constituencies that may be affected. For face recognition, differences in appearance, expression, and behavior may create problems when not represented in training and evaluation.
RISK ASSESSMENT OF EXISTING AI SYSTEMS FOR PWD
- The risk assessment groups AI by related functions and identifies disability constituencies that may be affected, but it is a non-exhaustive starting point given new technologies and disability’s long tail.Co-occurring disabilities also make inclusion challenging [73].Source: RISK ASSESSMENT OF EXISTING AI SYSTEMS… · p. 2
Computer Vision
- Computer vision fairness testing should consider facial and bodily differences, assistive equipment, and atypical motion that may affect system inputs.The passage identifies appearance, facial expressions, body size or proportions, and assistive equipment as relevant considerations.Source: Computer Vision · p. 2
Face Recognition
- Face recognition may fail for facial differences or nonconventional expressions, including those associated with Down syndrome, autism, stroke, Parkinson’s disease, and blindness.Blind users may also wear dark glasses or not orient their face toward the camera as expected.Source: Face Recognition · p. 2
Body Recognition
Body recognition can create access barriers and recognition errors for people with mobility, morphology, or movement differences. The risks range from unusable gesture input to missed or mischaracterized pedestrians.
Body Recognition
- Gesture recognition may be unreliable for differences in morphology, tremor, spastic motion, fatigue, and medication-related changes in motor symptoms.Performance may vary over time or across days; gesture systems can use computer vision or other sensors, with different pros and cons for populations.Source: Body Recognition · p. 2
- If body recognition is the only permitted interaction, people with no or severely restricted movement may be locked out of the technology.The passage gives ALS and quadriplegia as examples.Source: Body Recognition · p. 2
- Pedestrian detection may miss people with posture differences or wheelchair users, or misestimate the speed and trajectory of people who move differently than expected.The passage connects this risk to missing examples in training and evaluation.Source: Body Recognition · p. 2
Object, Scene, and Text Recognition
Object recognition and OCR systems are used by people with visual impairments, but common training images may differ from images captured by blind users or people with motor disabilities. Evaluation metrics may also miss end-user experience.
Object, Scene, and Text Recognition
- Object, scene, and text recognition systems are used by people with visual impairments, including through tools such as Seeing AI, Lookout, and KNFB Reader.These systems recognize objects, text, handwriting, and other visual content and output labels, captions, or properties.Source: Object, Scene, and Text Recognition · pp. 2–3
- Object-recognition errors often increase on images captured by blind users because their framing, blur, angles, and lighting differ from sighted-user training images.Similar problems may occur with photos taken by people with tremor or other motor disabilities; handwriting OCR may also struggle when motor disabilities affect neatness.Source: Object, Scene, and Text Recognition · p. 3
- Standard vision-system error metrics may not capture disabled users’ experience, especially when users such as blind people cannot independently verify system outputs.The passage notes that a blind user may have to rely on an object-detection system’s output.Source: Object, Scene, and Text Recognition · p. 3
Speech Systems
Speech systems can support access and communication, but disability-related differences may reduce their accuracy or usability, and some people may be unable to use speech recognition at all.
Speech Systems
- Speech systems recognize speech content or properties, or generate speech from symbolic inputs.Source: Speech Systems · p. 3
Speech Recognition
- ASR can provide captions for DHH users and speech input for people who have difficulty using their hands.Source: Speech Recognition · p. 3
- Atypical speech, disability-related accents, dysarthria, and speech-generating AAC can challenge ASR, while some people unable to speak may be locked out.Some older adults also face challenges related to pitch, pacing, and speech clarity; personalized models are possible for some groups.Source: Speech Recognition · p. 3
- Word Error Rate may not capture disabled users’ experience, especially when users such as profoundly deaf caption viewers cannot verify ASR output.Source: Speech Recognition · p. 3
Speech Generation
Speech generation serves assistive and communication needs, while speaker analysis and speech-generation defaults may fail to reflect some disabled users’ characteristics and preferences.
Speech Generation
- TTS supports screen readers and AAC, while personalized voice fonts may be valued by people with degenerative conditions such as ALS.Source: Speech Generation · p. 3
- Comprehensible speaking-rate defaults and comprehension metrics may need adjustment for disability segments with different rate needs.Cognitive or intellectual disabilities may require slower speech, whereas people with visual impairments may find rates too slow.Source: Speech Generation · p. 3
- AAC prediction corpora may need to reflect users’ topical needs and desired speech attributes to support expressivity and authentic self-representation.Source: Speech Generation · p. 3
Speaker Analysis
- Speaker analysis can support DHH sound awareness, but personal-characteristic or emotion inferences may fail for atypical speech or prosody.Examples include dysarthria, autism, and some types of dementia.Source: Speaker Analysis · p. 3
Text Processing
Text processing and analysis may be inaccurate or less useful for people with cognitive or intellectual disabilities, including in tools that can otherwise support reading and writing.
Text Processing
- Text-processing systems may have accuracy and fairness challenges for people with cognitive or intellectual disabilities, and minority languages used by disability subcommunities are also a concern.American Sign Language is given as an example.Source: Text Processing · p. 4
Text Analysis
- Text analysis can support reading and writing through visual highlighting and spelling, grammar, or word suggestions.Source: Text Analysis · p. 4
- Dyslexic spelling may challenge correction and query rewriting, while autistic expression of emotion in writing may be misclassified, affecting hiring or essay grading when such metrics are used.Source: Text Analysis · p. 4
Integrative AI
Integrative AI combines models into more complex systems; information retrieval may amplify bias or disadvantage disabled users through search and recommendation functions.
Integrative AI
- Integrative AI architectures combine several models to achieve more complex behavior; information retrieval and conversational agents are examples.Source: Integrative AI · p. 4
Information Retrieval
- Information retrieval uses functions such as query rewriting, autocomplete, ranking, summarization, and question answering across text, image, video, and sound.Source: Information Retrieval · p. 4
- IR may amplify bias through stereotyped or uneven search results, while advertising and recommender systems may differentially expose PWD to prices or opportunities.People with dyslexia have reported that standard query completion and result ranking may not match their abilities.Source: Information Retrieval · p. 4
Conversational Agents
Conversational agents may provide cognitive assistance, but their usefulness for people with disabilities depends on handling diverse communication needs. Other AI building blocks—including outlier detection, aggregate evaluation, objectives, and training data—can also create fairness risks.
Conversational Agents
- Conversational agents may reduce workload and assist people with dementia or intellectual disabilities, but poorly built agents can return stereotypes or provide poor user experiences.They may need to interpret atypical spelling, adjust vocabulary, or support sign, pictures, or icons.Source: Conversational Agents · p. 4
- Conversational agents should be trained and tested with diverse users and accommodate varied language and expressive media.Examples include dyslexic users’ atypical phrasing, vocabulary needs for people with dementia, sign languages, and pictures or icons.Source: Conversational Agents · p. 4
Other AI Techniques
- Underrepresented disabled users may be flagged as outliers when legitimate inputs differ from expected patterns, including atypical task-completion times.Examples include slower performance related to screen readers, motor limitations, or dyslexia.Source: Other AI Techniques · pp. 4–5
- Aggregate metrics can hide performance drops for small groups such as PWD, while aggregate objectives may fail to prioritize them.Some training objectives add terms penalizing performance discrepancies between subgroups [1].Source: Other AI Techniques · p. 5
- Existing datasets may omit diverse groups or real-world complexity, creating model blind spots; inclusive datasets can support both training and benchmark testing.The paper also identifies outlier detection, aggregate evaluation, and objective functions as building-block practices that may lead to bias against PWD.Source: Other AI Techniques · pp. 4–5
DISCUSSION
The discussion frames the paper as the first step of a broader roadmap: identify inclusion-related risks, then test hypotheses and develop datasets and techniques. It emphasizes that risks vary by harm type and that proposed hypotheses remain to be systematically tested.
DISCUSSION
- The paper mostly addresses roadmap step one: identifying how lack of inclusion in training and evaluation may harm fairness for PWD.The roadmap also proposes testing hypotheses, creating benchmark datasets, and developing modeling, mitigation, and error-measurement techniques.Source: DISCUSSION · p. 5
- Examples span service quality, allocation, denigration, stereotyping [7], and over- or under-representation; stereotyping and denigration may require qualitative investigation.Benchmarking objective fairness metrics may suffice for service quality, allocation, and representation issues.Source: DISCUSSION · p. 5
- Some concerns have cited evidence, while others are hypotheses based on domain knowledge and analogous errors for other minority groups; systematic testing remains necessary.The authors’ cautious wording reflects uncertainty about these proposed risks.Source: DISCUSSION · p. 5
- Creating public datasets for testing and benchmarking raises ethical questions about privacy, labeling, rare conditions, intersectionality, accessibility, consent, and coverage.The paper identifies these dataset questions as future work for vulnerable user groups.Source: DISCUSSION · p. 5
- If existing modeling, mitigation, or measurement methods prove inadequate, research may need to weigh general models against personalization and involve PWD in defining scenarios, metrics, and policies.Personalized training may add barriers and risk a two-tier system in which general models work by default for some people but not others.Source: DISCUSSION · p. 5
CONCLUSION
The conclusion presents the paper as a research roadmap for improving AI efficacy and fairness for people with disabilities while avoiding potential harms.
CONCLUSION
- The paper reflects on how AI systems and their building-block techniques may limit efficacy and fairness for PWD, and calls for guidelines, datasets, algorithms, and error metrics.The authors hope the roadmap will guide researchers and practitioners toward systems that are fair to and effective for PWD.Source: CONCLUSION · p. 5