Source-linked AI summary
Does "AI" stand for augmenting inequality in the era of covid-19 healthcare?
David Leslie, Anjali Mazumder, Aidan Peppin, Maria Wolters, Alexa Hagerty
TL;DR
Covid-19 has disproportionately harmed disadvantaged communities, while AI systems used against the pandemic may compound those inequities. The paper examines bias across datasets, data representation, and human choices throughout the AI lifecycle, concluding that innovation must be tempered by attention to inequality and discrimination.
Problem
Covid-19 has disproportionately affected disadvantaged communities, while AI technologies used to combat the pandemic are susceptible to biases that may augment existing inequality.
Method
The paper analyzes three sources of AI bias: health discrimination entrenched in datasets, inadequate data representativeness, and human choices across design, development, and deployment.
Results
AI use threatens to exacerbate covid-19's disparate effects on marginalised, under-represented, and vulnerable groups.
Takeaways & Limitations
Decision makers, technology developers, and health officials must account for potential biases and inequities at all stages of the AI process.
Takeaways & Limitations
The conclusion is bounded by existing health inequalities, disproportionate pandemic vulnerability, and sociotechnical determinants of algorithmic discrimination.
Abstract
from arXiv · showhide
Among the most damaging characteristics of the covid-19 pandemic has been its disproportionate effect on disadvantaged communities. As the outbreak has spread globally, factors such as systemic racism, marginalisation, and structural inequality have created path dependencies that have led to poor health outcomes. These social determinants of infectious disease and vulnerability to disaster have converged to affect already disadvantaged communities with higher levels of economic instability, disease exposure, infection severity, and death. Artificial intelligence (AI) technologies are an important part of the health informatics toolkit used to fight contagious disease. AI is well known, however, to be susceptible to algorithmic biases that can entrench and augment existing inequality. Uncritically deploying AI in the fight against covid-19 thus risks amplifying the pandemic's adverse effects on vulnerable groups, exacerbating health inequity. In this paper, we claim that AI systems can introduce or reflect bias and discrimination in three ways: in patterns of health discrimination that become entrenched in datasets, in data representativeness, and in human choices made during the design, development, and deployment of these systems. We highlight how the use of AI technologies threaten to exacerbate the disparate effect of covid-19 on marginalised, under-represented, and vulnerable groups, particularly black, Asian, and other minoritised ethnic people, older populations, and those of lower socioeconomic status. We conclude that, to mitigate the compounding effects of AI on inequalities associated with covid-19, decision makers, technology developers, and health officials must account for the potential biases and inequities at all stages of the AI process.
Embedding inequality in AI systems
AI systems can embed inequality across conception, design, and use, including through discriminatory training data, unrepresentative samples, and development choices that produce disparate subgroup performance.
- Embedding inequality in AI systems: AI systems can entrench health inequality across conception, design, and use.
- Embedding inequality in AI systems: Training datasets can encode discriminatory structures when underserved communities are excluded from healthcare data.
- Embedding inequality in AI systems: Unrepresentative data can undersample vulnerable populations.
- Embedding inequality in AI systems: Development and implementation choices can create disparate performance for vulnerable subgroups.
Health discrimination in datasets
AI systems can reproduce health discrimination when clinical datasets reflect biased practices, notes, or healthcare access disparities. These inherited patterns may be repeated and compounded when models are used during covid-19.
- Health discrimination in datasets: Clinical datasets can carry discriminatory judgments and healthcare processes into electronic records, notes, trials, and public-health monitoring.
- Health discrimination in datasets: AI models can silently track real-world biases embedded in existing practices and processing technologies.
- Health discrimination in datasets: Models trained on biased medical data may incorporate inequitable practices and reinforce discriminatory structures.
- Health discrimination in datasets: During covid-19, biased clinical notes or incomplete electronic records may cause AI systems to reflect, repeat, and compound structural discrimination.
Data representativeness
AI models may be less reliable for disadvantaged groups when training data under-represent populations, healthcare access, or digital participation. Covid-19 data can also overrepresent patients from privileged hospital catchments.
- Data representativeness: Datasets often under-represent or exclude people without digital access, including minoritised ethnicities, immigrants, and socioeconomically disadvantaged groups.
- Data representativeness: In the UK, more than 20% of people aged 15 or older lack essential digital skills, and up to 10% of some subgroups do not own smartphones.
- Data representativeness: Applying models tailored to dominant groups in a one-size-fits-all manner may yield poorer performance for disadvantaged groups.
- Data representativeness: Covid-19 electronic health-record datasets may overrepresent patients with access to particular hospitals in well-off neighbourhoods.
- Data representativeness: Without addressing dataset imbalances and model limitations, resulting models may not generalise and may underfit vulnerable groups.
Biases in the choices made for AI design and use
Human choices can introduce bias throughout AI agenda setting, problem formulation, data handling, model construction, and implementation. These choices can mismeasure illness, encode structural racism, or shape how clinicians use recommendations.
- Biases in the choices made for AI design and use: Research agendas may exclude affected communities because decision makers do not reflect their sociodemographic composition.
- Biases in the choices made for AI design and use: Urgency and institutional hierarchies during covid-19 can conflict with consensus building, oversight, and community involvement in setting agendas.
- Biases in the choices made for AI design and use: Target-variable and proxy choices can fail to capture the social contexts of discrimination and introduce structural inequalities into models.
- Biases in the choices made for AI design and use: Using healthcare costs as a proxy for illness systematically mismeasured chronic illness among millions of African Americans.
- Biases in the choices made for AI design and use: Feature inclusion and preprocessing choices can cause models to treat socioeconomic or environmental origins as biological characteristics.
- Biases in the choices made for AI design and use: Clinicians may over-rely on faulty AI recommendations or discount corrections to discrimination because of their own professional preconceptions.
Equity under pressure
Urgency during the covid-19 pandemic encouraged rapid development and repurposing of AI systems, but rushed or mismatched applications risked unreliable and discriminatory effects for vulnerable groups.
- Equity under pressure: Rapid-response pressure and swift repurposing of AI systems can hinder responsible design and use during the pandemic.The paper links this urgency to the risks created by insufficient validation and unrepresentative data.
- Equity under pressure: Rushed covid-19 prediction models showed high risks of statistical bias, poor reporting, and overoptimistic performance.The cited review recommended that none of the models be used in medical practice at that point.
- Equity under pressure: Repurposed AI systems trained outside the pandemic context were used for sensitive tasks such as predicting infected patients’ need for intensive care or ventilation.These systems faced risks of insufficient validation, inconsistent reliability, and poor generalisability.
- Equity under pressure: A mismatch between training populations and groups disproportionately affected by covid-19 can undermine model reliability and generalisability.Unrepresentative samples were identified as a source of these risks.
Conclusion
The paper argues that AI can contribute to covid-19 healthcare only when development and deployment address existing inequalities and algorithmic bias. It calls for end-to-end, inclusive safeguards so AI counters rather than compounds health inequities.
- Conclusion: AI’s potential contribution to clinical, research, and public health tools depends on addressing inequalities and algorithmic bias.The paper contrasts this approach with AI contributing to existing inequalities.
- Conclusion: Responsible covid-19 AI requires deliberate, end-to-end bias detection and mitigation embedded throughout technological development.The proposed process includes clinical expertise, inclusive community involvement, interdisciplinary knowledge, and ethical reflexivity.
- Conclusion: AI projects should integrate social determinants of disparate covid-19 vulnerability into data gathering and combine socioeconomic information with race, ethnicity, and other sensitive data.The paper also emphasizes individual and community consent when determining the purpose and path of innovation projects.
- Conclusion: Systemic racism, wealth disparities, and other structural inequities are identified as root causes of discrimination and health inequalities.Addressing these broader conditions is presented alongside responsible AI design and deployment.
- Conclusion: Decision makers, technology developers, and health officials must account for bias and inequity at every stage of the AI process.The paper highlights marginalised, under-represented, and vulnerable groups, including minoritised ethnic people, older populations, and people with lower socioeconomic status.