Source-linked AI summary
Ethical Machine Learning in Health Care
Irene Y. Chen, Emma Pierson, Sherri Rose, Shalmali Joshi, Kadija Ferryman, Marzyeh Ghassemi
TL;DR
Ethical concerns arise because health care machine-learning models can worsen existing inequities. This paper frames equitable machine learning through social justice, proposes an ethical pipeline, and identifies challenges and recommendations from problem selection through deployment.
Problem
Health care machine-learning models can amplify existing inequities, but ethical considerations for equitable development remain insufficiently organized across the modeling pipeline.
Method
The paper frames health care machine-learning ethics through social justice and proposes a five-step pipeline spanning problem selection through post-deployment.
Results
The paper identifies technical and ethical challenges at each pipeline stage and summarizes recommendations for addressing them.
Takeaways & Limitations
Equitable health care machine learning requires attention to social justice throughout model development and deployment.
Takeaways & Limitations
Generalizing randomized-trial treatment effects to other populations can be challenging with available data and may require strong assumptions.
Abstract
from arXiv · showhide
The use of machine learning (ML) in health care raises numerous ethical concerns, especially as models can amplify existing health inequities. Here, we outline ethical considerations for equitable ML in the advancement of health care. Specifically, we frame ethics of ML in health care through the lens of social justice. We describe ongoing efforts and outline challenges in a proposed pipeline of ethical ML in health, ranging from problem selection to post-deployment considerations. We close by summarizing recommendations to address these challenges.
1. INTRODUCTION
This review frames ethical machine learning in health care through social justice, focusing on how the model-development pipeline can exacerbate inequities. It examines ethical considerations from problem selection through post-deployment, using Black maternal mortality to illustrate how inequity can arise across stages.
- Machine learning in health care may exacerbate existing health disparities, including through underperformance on women and ethnic and racial minorities.
- Ethical guidance at the intersection of health, machine learning, and fairness remains comparatively underdeveloped.
- The proposed five-step ethical pipeline links technical challenges in model development to justice violations that may worsen existing or create new social injustices.
- The review focuses on equity-related differences between groups induced by or related to the model-development pipeline, drawing on bioethics and public health ethics.
- Black women in the United States die in childbirth at a rate three times higher than white women, illustrating how inequity can arise across pipeline stages.The review identifies problem selection, hospital data quality and quantity, outcome definitions, and algorithm development as relevant stages.
- The review proceeds sequentially from problem selection and funding through data collection, outcome definition, algorithm building, and post-deployment.
2. PROBLEM SELECTION
Problem selection is a matter of justice because research questions and funding priorities can focus on advantaged groups and exacerbate socioeconomic, racial, and gender injustices. Evidence points to global, racial, gender, and workforce-related biases in which health problems are studied and funded.
- 2. PROBLEM SELECTION: Research priorities can reproduce injustice when proposed and funded questions focus on the health needs of advantaged groups.The paper identifies socioeconomic, racial, and gender disparities in research teams and funding priorities as mechanisms that exacerbate existing injustices.
- 2. PROBLEM SELECTION: 14% of the global disease burden is attributable to 26 poverty-related diseases, but they receive only 1.3% of global health-research funding.This exemplifies the “10/90” gap, in which health-research dollars disproportionately address problems affecting a small fraction of the global population.
- 2. PROBLEM SELECTION: Cystic fibrosis receives 3.4 times more US NIH funding per affected individual than sickle cell disease, despite similar severity.Sickle cell disease is more common in Black patients, whereas cystic fibrosis is more common in white patients.
- 2. PROBLEM SELECTION: Estimates of endometriosis prevalence range from 1% to 10%, reflecting how women’s health conditions remain poorly understood and understudied.The paper also identifies menstrual-cycle stigma and limited research as undermining understanding of the health of half the global population.
- 2. PROBLEM SELECTION: Diversifying the scientific workforce can produce problem selection that more equitably represents the population’s interests and needs.Scientists from underrepresented racial and gender groups prioritize different topics and produce more novel research, although their innovations are taken up at lower rates.
3. DATA COLLECTION
Health data are increasingly central to health policy and machine learning, but collection processes systematically introduce missingness, noise, and population-specific bias. These limitations can exclude or misrepresent marginalized communities, undermining the representativeness and equity of health-related models and decisions.
- Types of Data Loss: Data collection produces heterogeneous data loss across input types and population-specific data loss affecting whether particular individuals’ information is collected.Examples include restrictive clinical-trial criteria and social media data reflecting people with device access.
- Randomized Controlled Trials: 94% of the adult asthmatic population would not have been eligible for trials used to define asthma treatment.Although methods seek to generalize RCT treatment effects, empirical evidence indicates generalization can be difficult or require strong assumptions.
- Electronic Health Records: EHR datasets reflect patient health, health care systems, providers, access to care, and available systems, producing meaningful missingness and population bias.MIMIC-III is largely based on intensive-care patients at BIDMC, which has a largely white patient population.
- Social Media Data: Social media data exclude people without internet access and are further shaped by platform choices, geography, and nonrandom collection pipelines.Restricting samples to desktop rather than mobile users can be problematic in non-North American contexts.
- Underserved Populations: Historically underserved groups are underrepresented, misrepresented, or missing from health data, while genomics datasets often yield more accurate genetic risk scores for European-ancestry individuals.Affected populations include low- and middle-income nationals, transgender and gender non-conforming individuals, undocumented migrants, and pregnant women.
- Low- and Middle-Income Nationals: 9 out of 46 member states in Sub-Saharan Africa could produce death statistics for a global burden-of-disease assessment in 2001.Health data in low- and middle-income nations are often infrequently collected, not digitized, or erroneous.
- Pregnant Women: The US maternal mortality rate more than doubled from 9.8 per 100,000 live births in 2000 to 21.5 in 2014.Confidentiality protocols may suppress low-count information in nationally available datasets, affecting data visibility for pregnant women.
4. OUTCOME DEFINITION
Defining health care outcomes requires reliable proxies and attention to label noise because prevalence, care access, clinical practices, and socioeconomic factors can distort outcomes. These choices can substantially affect the performance and equity of resulting models, including when health care costs are used as a proxy for health need.
- Outcome definition: Outcome definitions can be skewed by disease prevalence and differences in how diseases manifest across patient populations.
- Outcome definition: EHR-based labels may reflect unequal care access, economic incentives, clinical protocols, racial mistrust, and inconsistent data collection rather than disease status alone.
- Outcome definition: Developers should choose reliable proxies and account for outcome-label noise because these choices can strongly affect model performance and equity.
- Clinical diagnosis: Clinical diagnosis labels vary by definition, while billing-oriented diagnosis codes create ambiguity and can be under-reported or over-reported across facilities.
- Health care costs: Health care costs can diverge from individual health need because of socioeconomic factors, and using costs to allocate care produced racial disparities.
- Health care costs: When health need is the priority, developers can select alternatives such as chronic-disease counts instead of health care costs.
5. ALGORITHM DEVELOPMENT
Algorithm development is a major ethical vulnerability and opportunity because choices about losses, features, tuning, metrics, and fairness encode values and can affect equity, reliability, and deployment. Ethical development therefore requires contextual understanding of confounding and sensitive attributes, careful feature selection and tuning, use-case-appropriate evaluation, and explicit fairness definitions.
- Algorithm development: Algorithmic choices such as the loss function are not purely technical: surrogate losses can disproportionately affect undersampled groups, while optimizing worst-case rather than average error reflects value judgments.Research teams control these choices after determining the research questions, creating both ethical vulnerability and opportunity.
- Confounding: Confounding features can produce spurious correlations that fail on test data, and simply adding them to models may be insufficient when they mediate or moderate outcomes.Hair-color classifiers trained on confounded data learned gender-biased decision boundaries; clinical models likewise require careful treatment of confounding.
- Feature selection: Blindly incorporating race and gender can exacerbate inequities in diagnostics and treatments, embedding bias in algorithms used for clinical care.Sensitive attributes may also be misclassified or missing, so their availability does not eliminate the need for contextual understanding.
- Feature selection: Automated feature selection remains context-dependent: stepwise regression can depend on feature-processing order and misleading p-values, while lasso trades potential interpretability for selection.The passage identifies weaknesses in global-fit-based selection and emphasizes that automation does not replace contextual understanding.
- Tuning parameters: Tuning choices can overfit training data and reduce generalizability, as default random-forest settings may grow trees until terminal leaves contain one observation.Tuning parameters may be set a priori or selected through cross-validation, including neural-network learning rates and random-forest leaf sizes.
- Fairness and evaluation: Fairness definitions shape loss-function and algorithm choices, while evaluation metrics must match the use case and the relative value of true and false decisions.The paper distinguishes individual fairness from group fairness and notes that AUC can mask weak true-positive performance.
6. POST-DEPLOYMENT CONSIDERATIONS
Post-deployment ethical ML requires careful performance reporting, auditing for bias and clinical impact, evaluation of generalizability, and documentation of unintended consequences. Regulatory frameworks should address model drift, health disparities, and health equity audits during postmarket evaluation.
- Auditing: Regular post-deployment audits should inspect model performance across groups and outcomes to identify bias and downstream clinical impacts.Audits can reveal how models affect different populations and identify areas of potential concern.
- Generalizability: Distribution shifts between development and deployment settings can substantially reduce performance, making model generalizability a crucial deployment concern.Chest X-ray models may perform well on same-hospital test data but degrade rapidly on data from another hospital.
- Documentation: Clear documentation should record clinically specific development features, clinical logistics, potential unintended consequences, and bias–performance trade-offs.Co-designing checklists with clinical practitioners can formalize deployment considerations.
- Regulation: Models that continue learning from observed outcomes face additional model-drift risks and may require additional audits compared with frozen models.Regulatory frameworks should distinguish between models trained and then frozen and models that continue learning after deployment.
- Regulation: Health care ML regulation should explicitly address health disparities across development stages and include health equity audits in postmarket evaluation.Proposed frameworks could also assess patient benefit against standard clinical practice and define protocols for adverse events and model recalls.
7. RECOMMENDATIONS
The recommendations call for equity-focused problem selection, data collection, outcome choice, and explicit reflection on model goals throughout the ML development pipeline. They also argue that ML should be harnessed to shift power toward underserved patients and preserve patient autonomy.
- Pipeline-wide ethics: Hidden challenges such as existing health inequities and outcome-label bias can remain unaddressed in typical ML health projects despite serious ethical consequences.The review contrasts these hidden challenges with more commonly recognized concerns such as task accuracy and outcome distribution shift.
- Problem selection: Diverse teams and equity-oriented frameworks should guide problem selection, including attention to historically understudied problems.Such problems are identified as important targets for practitioners seeking high-impact work.
- Data collection: Data collection should be treated as a front-of-mind pipeline concern, with disclosures about dataset imbalance and input from domain experts.Domain experts can help ensure that data reflect the needs of underserved and understudied populations.
- Outcome choice: Outcome choice should match the task and preferably avoid bias, while ethically biased labels require accounting for inequity in model design.Recommended responses include bias-removal approaches during pre-processing or use of a reasonable proxy.
- Model goals: Researchers should reflect on model goals during development and articulate that reflection in a pre-analysis plan.The recommendation makes goal reflection an essential part of development rather than an afterthought.
- Power and deployment: ML should be harnessed to shift power in health care systems by benefiting underserved patients, improving access, and preserving patient autonomy.Suggested applications include selecting beneficial problems, targeting systemic interventions, and enforcing evaluations explicitly aimed at autonomy.
FUTURE QUESTIONS
The section identifies future questions about addressing health injustices during global crises, embedding ethics throughout ML development, responding to injustices discovered after clinical impact, and translating evaluation and audits into clinical practice.
- FUTURE QUESTIONS: How can global health crises that worsen existing health injustices be addressed?
- FUTURE QUESTIONS: How can model developers incorporate ethical considerations into the ML pipeline from the outset?
- FUTURE QUESTIONS: What should developers do when egregious injustices are discovered only after clinical impact has occurred?
- FUTURE QUESTIONS: How can ML evaluation and audits be translated into meaningful clinical practice across countries?
DISCLOSURE STATEMENT · LITERATURE CITED
The authors report no known affiliations, memberships, funding, or financial holdings that could be perceived as affecting the review’s objectivity.
- DISCLOSURE STATEMENT: The authors report no affiliations, memberships, funding, or financial holdings known to affect the review’s objectivity.The statement concerns potential perceived effects on objectivity.