PaperlayerView exact revision on arXiv

Source-linked AI summary

Machine learning and AI research for Patient Benefit: 20 Critical Questions on Transparency, Replicability, Ethics and Effectiveness

Sebastian Vollmer, Bilal A. Mateen, Gergo Bohner, Franz J Király, Rayid Ghani, Pall Jonsson, Sarah Cumbers, Adrian Jonas, Katherine S. L. McAllister, Puja Myles, David Granger, Mark Birse, Richard Branson, Karel GM Moons, Gary S Collins, John P. A. Ioannidis, Chris Holmes, Harry Hemingway

arXiv:1812.10404v1cs.CYcs.LGstat.APstat.ML

TL;DR

Health-related ML/AI applications have highlighted fundamental issues concerning effectiveness and impact on health. The paper proposes a framework intended to support safe and effective ML/AI-based research and robust decision making at every stage. The proposed questions provide an initial framework for transparent, replicable, ethical, and effective health-related ML/AI research.

Abstract

from arXiv · show

Machine learning (ML), artificial intelligence (AI) and other modern statistical methods are providing new opportunities to operationalize previously untapped and rapidly growing sources of data for patient benefit. Whilst there is a lot of promising research currently being undertaken, the literature as a whole lacks: transparency; clear reporting to facilitate replicability; exploration for potential ethical concerns; and, clear demonstrations of effectiveness. There are many reasons for why these issues exist, but one of the most important that we provide a preliminary solution for here is the current lack of ML/AI- specific best practice guidance. Although there is no consensus on what best practice looks in this field, we believe that interdisciplinary groups pursuing research and impact projects in the ML/AI for health domain would benefit from answering a series of questions based on the important issues that exist when undertaking work of this nature. Here we present 20 questions that span the entire project life cycle, from inception, data analysis, and model evaluation, to implementation, as a means to facilitate project planning and post-hoc (structured) independent evaluation. By beginning to answer these questions in different settings, we can start to understand what constitutes a good answer, and we expect that the resulting discussion will be central to developing an international consensus framework for transparent, replicable, ethical and effective research in artificial intelligence (AI-TREE) for health.

The Need for Guidance Today

Health-related ML/AI research lacks guidance for distinguishing good from bad practice across development, reporting, evaluation, and clinical impact. The paper proposes 20 critical questions organized around transparency, replicability, ethics, and effectiveness to support planning and independent appraisal.

  • Few health-related ML/AI algorithms have conclusively demonstrated improved patient outcomes, making translation from the computer-bench to the bedside difficult to assess.Source: The Need for Guidance Today · p. 4

The Need for Guidance Today

  • The framework addresses a lack of guidance across the healthcare ML/AI pipeline, from design and data analysis through reporting and effectiveness evaluation.Source: The Need for Guidance Today · p. 4

The Need for Guidance Today

  • The questions span inception, data analysis, impact evaluation, and implementation, including patient benefit, data suitability, reproducibility, equity, effectiveness, monitoring, cost, commercialization, and regulation.Source: The Need for Guidance Today · p. 7

1. How is the ML/AI model embedded in feedback loops to facilitate a learning health system?

The paper places ML/AI research within a learning health system, where data science informs decisions through iterative feedback rather than a linear path to an end user. It also emphasizes a clearly specified patient-benefit question and patient and public involvement throughout the data-science cycle.

1. How is the ML/AI model embedded in feedback loops to facilitate a learning health system?

  • A learning health system uses data science to inform clinical decision making and health-system operations in an iterative feedback loop.Source: §1 · p. 8
  • Researchers should describe which parts of the healthcare data-science cycle their work engages with, including when a project is only proof of concept.Source: §1 · p. 8

2. What is the health question relating to patient benefit?

  • A health question should specify the clinical decision, consequent action, patient outcome, trade-offs, and likelihood of successful deployment addressed by the model.Source: §2 · p. 8

3. When and how should patients be involved in data collection, analysis, deployment, and use?

  • Patient and public involvement should extend across identifying the need for an algorithm, supporting its development, and determining acceptability in practice.Source: §3 · p. 8

4. Is there organisational transparency about the flow of data?

Trustworthy health-related ML/AI requires transparency about how data move among organizations and whether available data can answer the clinical question. The paper therefore links organizational disclosure with explicit checks of data relevance, quality, representativeness, and behavior outside the training range.

4. Is there organisational transparency about the flow of data?

  • Legal compliance alone is insufficient: trustworthy ML/AI research requires transparency about institutional interactions and data access, transfer, and storage.Source: §4 · p. 9
  • Clear declarations of interest and explanations of interactions among public, nonprofit, industrial, and internal organizations can support trust in models.Source: §4 · p. 9

heterogeneity, and is it of sufficient detail and quality?

6. Does the validation methodology reflect the real-world constraints and operational procedures

Validation and data-access practices must reflect how healthcare data are generated, stored, and accessed in practice. The paper highlights temporal and cross-site distribution changes, reproducible access mechanisms, and computational and licensing constraints as important boundaries.

associated with data collection and storage?

7. On what basis are data accessible to other researchers?

  • Data-sharing plans should provide clear access mechanisms for replication and validation, using open, restricted, remote, or enclave-based arrangements where appropriate.Source: §7 · p. 10

problem?

  • Computational resources can limit feasible analyses, while software licensing can affect an algorithm's commercial potential and future use.Source: problem? · pp. 10–11

used?

Performance metrics must reflect the clinical decision and patient outcomes, while reported statistical gains must be justified against implementation costs and alternatives.

used?

  • Misclassification error or vaguely related metrics may fail to demonstrate improved patient outcomes, especially in imbalanced screening applications.The example of WFO illustrates the importance of reporting clinically relevant performance measures alongside statistical measures.Source: used? · p. 11
  • Metric selection should reflect both the statistical task and the clinical decision-making process to which results will be applied.Relevant parties, including patients, data scientists, and clinicians, should be consulted.Source: used? · p. 11

context of any trade-offs?

  • A clinically meaningful advantage is necessary but must be weighed against the costs of development, deployment, maintenance, additional data collection, and reduced interpretability.The trade-off is situation specific and may include the need for subsidiary models to recover inferential capabilities.Source: context of any trade-offs? · pp. 11–12
  • AUC improvements of 0.01 to 0.02 across mortality, readmission, and length-of-stay tasks illustrate why statistical gains require clinical justification.The comparison was between a deep learning model and logistic regression.Source: context of any trade-offs? · p. 11

11. Is the ML/AI algorithm compared to the current best technology, and against other appropriate

AI/ML systems should be compared with accepted standards, existing models, and null alternatives, while transparent reporting and external validation are needed to assess reproducibility and generalizability.

baselines?

  • AI/ML models should be compared with accepted statistical approaches, published models, and null models rather than weak competitors or no comparator.Avoiding appropriate comparisons can exaggerate apparent benefit through weak-comparator or strawman bias.Source: baselines? · p. 12
  • Design flaws and reporting weaknesses in proposed Framingham-score modifications cast doubt on claims of improved prediction.This example underscores the need to assess algorithmic claims against established approaches.Source: baselines? · p. 12

(including the methods, parameters, random seeds, etc. utilised)?

  • In 28 MIMIC-based studies, off-the-shelf logistic regression and boosted trees outperformed the reported ML/AI model in 64% and 82% of replications, respectively.More than half of reproduced sample sizes differed by over 25% because inclusion and exclusion criteria were insufficiently described.Source: (including the methods, parameters,… · p. 12
  • Providing code, raw data where possible, and detailed preprocessing and population definitions is central to enabling replication.Code-sharing and curation platforms such as GitHub are identified as potential solutions.Source: (including the methods, parameters,… · pp. 12–13

13. Are the results reproducible in settings beyond where the system was developed (i.e. external validity

  • External validation is relatively scarce, and predictive accuracy commonly declines relative to development-study performance in new settings.The extent to which this reflects inadequate reporting or other issues remains unclear.Source: §13 · p. 13

sex, ethnicity or other protected characteristics?

Fairness evaluation should examine subgroup performance and representation throughout the data science cycle, while interpretability and clinical adoption depend on the intervention, decision, and affected users.

sex, ethnicity or other protected characteristics?

  • Bias can arise when key populations are underrepresented, producing different predictive accuracy and potentially exacerbating inequities across subgroups.Fairness considerations therefore apply from data collection through implementation.Source: sex, ethnicity or other protected… · p. 13
  • Fairness analysis should compare algorithmic bias with existing-system bias and match investigated errors to the consequences of resulting interventions.False positives matter particularly when interventions are costly or harmful, whereas false negatives matter for predominantly assistive interventions.Source: sex, ethnicity or other protected… · pp. 13–14
  • Developers should demonstrate that sensible decision thresholds do not create or exacerbate inequalities, even when threshold setting is outside their remit.Methodological tools for fairness evaluation can support this analysis [72,73].Source: sex, ethnicity or other protected… · p. 14

interpretable?

  • Clinical adoption depends on usefulness and the interpretability lost when moving from white-box to black-box models.Explanation requirements increase for decisions such as allocating potentially life-prolonging treatments.Source: interpretable? · p. 14
  • Models whose outputs do not directly answer a specific clinical question, or whose processing is difficult to explain and justify, limit usefulness and trust despite robust statistical gains.Interpretability can only be determined through consultation and collaboration with clinicians and patients.Source: interpretable? · p. 14

unintended consequences prevented?

Real-world evaluation must test unintended consequences and intended use, then continue monitoring because effectiveness and utility can change as clinical environments evolve.

unintended consequences prevented?

  • Only a couple of hundred randomized clinical trials among almost a million examined how diagnostic tests affected clinical outcomes.This illustrates the limited evidence base for clinical utility.Source: unintended consequences prevented? · p. 14
  • Patient testing found concern that users might game Babylon Health’s symptom checker to obtain a general-practitioner appointment.Such unintended use supports pragmatic clinical trials to assess real-world deployment.Source: unintended consequences prevented? · pp. 14–15
  • Algorithms may lack regulatory incentives to demonstrate real-world effectiveness when they do not qualify as medical devices, exposing patients and health systems to unintended consequences.The paper links this gap to the need for careful validation and verification in routine clinical practice.Source: unintended consequences prevented? · pp. 14–15

changes (i.e. post-license monitoring)?

  • Registry studies found no significant diagnostic-accuracy improvement from mammography CAD, while incorrect prompts reduced discerning users’ sensitivity by up to 0.145 for difficult cases.These findings show that deployment effects can differ from initial efficacy evidence.Source: changes (i.e. post-license monitoring)? · p. 15
  • Healthcare technologies require constant review because their usefulness can change with population case-mix, eligibility, and treatment changes.The paper uses mammography CAD as an example of why post-license monitoring remains necessary.Source: changes (i.e. post-license monitoring)? · p. 15

18. Is the ML/AI model cost-effective to build, implement, and maintain?

The paper asks whether ML/AI models are cost-effective, how commercialization benefits should be distributed, and whether regulatory requirements have been addressed. It emphasizes evaluating clinical benefit, economic impact, stakeholder contributions, and changing regulatory obligations.

18. Is the ML/AI model cost-effective to build, implement, and maintain?

  • Costs arise from software development, data generation and use, implementation, and acting on additional information.Source: §18 · p. 15
  • Healthcare appraisal should consider a model’s clinical benefit over current practice alongside its costs or savings.Assessment frameworks should incorporate evidence of effectiveness and economic impact.Source: §18 · pp. 15–16

19. How will the potential financial benefits be distributed if the ML/AI model is commercialized?

  • In publicly funded health systems, retaining some benefits from data use helps satisfy the social contract with the public.Source: §19 · p. 16
  • Commercialization should recognize health-system and institutional contributions and establish mechanisms to compensate them.Contributions may include data acquisition, annotation, and publicly funded development time.Source: §19 · p. 16

20. How have the regulatory requirements for accreditation/approval been addressed?

  • Regulatory compliance must be reviewed regularly because requirements for health-related ML/AI software are changing dynamically.Regulatory discussions should begin early in product planning, but a single discussion is insufficient.Source: §20 · p. 16

Conclusion: from critical questions to a consensus TREE framework

The conclusion presents the questions as an initial step toward an international consensus framework for robust, transparent, and effective ML/AI decision-making. It identifies substantial challenges in developing actionable guidance while preserving innovation and transparency.

Conclusion: from critical questions to a consensus TREE framework

Declaration of Interests

The paper reports declarations of interest and funding from academic, governmental, regulatory, and industry-linked organizations. The listed support includes institutional grants, public research funding, and unrestricted gifts.

Declaration of Interests

Funding

  • Funding includes support from The Alan Turing Institute, NIHR, EPSRC, Horizon 2020, EFPIA, NICE, MHRA, and other research programs.Additional support includes the BigData@Heart Consortium, Stanford-linked gifts and grants, and University of Warwick funding.Source: Funding · p. 19
Loading 1812.10404v1…