PaperlayerView exact revision on arXiv

Source-linked AI summary

BIAS: Transparent reporting of biomedical image analysis challenges

Lena Maier-Hein, Annika Reinke, Michal Kozubek, Anne L. Martel, Tal Arbel, Matthias Eisenmann, Allan Hanbuary, Pierre Jannin, Henning Müller, Sinan Onogur, Julio Saez-Rodriguez, Bram van Ginneken, Annette Kopp-Schneider, Bennett Landman

arXiv:1910.04071v5cs.CVcs.CYeess.IV

TL;DR

Biomedical image analysis challenges attract attention, but their reporting has often been insufficient for interpreting and reproducing results. The BIAS initiative developed reporting recommendations and a checklist through prior challenge analyses and structured submissions. The resulting guidance is intended to make challenge information explicit and support review, while its authors note that the checklist may need future updating.

Abstract

from arXiv · show

The number of biomedical image analysis challenges organized per year is steadily increasing. These international competitions have the purpose of benchmarking algorithms on common data sets, typically to identify the best method for a given problem. Recent research, however, revealed that common practice related to challenge reporting does not allow for adequate interpretation and reproducibility of results. To address the discrepancy between the impact of challenges and the quality (control), the Biomedical I mage Analysis ChallengeS (BIAS) initiative developed a set of recommendations for the reporting of challenges. The BIAS statement aims to improve the transparency of the reporting of a biomedical image analysis challenge regardless of field of application, image modality or task category assessed. This article describes how the BIAS statement was developed and presents a checklist which authors of biomedical image analysis challenges are encouraged to include in their submission when giving a paper on a challenge into review. The purpose of the checklist is to standardize and facilitate the review process and raise interpretability and reproducibility of challenge results by making relevant information explicit.

I. INTRODUCTION

Biomedical image analysis challenges have high visibility, but their reporting and design practices have been found inadequate for interpreting and reproducing results. The BIAS initiative presents reporting guidance and a checklist to make relevant information explicit and support review.

I. INTRODUCTION

  • A comprehensive analysis found poor challenge reporting, heterogeneous design, and rankings sensitive to design choices, limiting interpretation and reproducibility.The reported sensitivities include metric variant, test-case aggregation, and who annotates the data.Source: §I · p. 1
  • The BIAS initiative developed a reporting guideline to standardize challenge-paper writing and review and make results easier to interpret and reproduce.Authors are encouraged to submit the checklist with challenge papers for review.Source: §I · p. 2
  • The checklist is guidance rather than a restriction on how authors arrange information; reviewers may request additions when relevant information is missing.Missing information can be marked “n/a” in the checklist’s page-number column.Source: §I · p. 2
  • The paper introduces challenge terminology and guideline-development methods, then recommends how to report challenge design and results before discussing the guidance.Source: §I · p. 2

II. METHODS

The authors draw on prior challenge analyses, prospective structured submissions, and co-author feedback to develop recommendations and a reviewer checklist for reporting biomedical image analysis challenges.

II. METHODS

  • The authors define a challenge as an open competition on a specific biomedical image analysis problem; a challenge can include multiple tasks with separate rankings.A case is a dataset on which a participating algorithm produces one result, and each case includes at least one biomedical image.Source: §II · p. 2
  • The guideline distinguishes metrics, reference annotations, and training, validation, and test phases when describing challenge assessment and participation.Reference annotations are approximations of the unknown truth, and challenge rankings are commonly based on test-case metric values.Source: §II · p. 2
  • The prior challenge analysis compiled 53 parameters and identified 150 biomedical image analysis challenges comprising 549 tasks over 12 years.The parameters covered organization, participation, mission, datasets, assessment, and outcomes.Source: §II · pp. 2–3
  • 62% was the median share of 53 relevant challenge parameters covered in reports, with crucial details such as metric aggregation and reference annotation often omitted.The analysis also found challenge design to be highly heterogeneous.Source: §II · p. 3
  • For MICCAI 2018/2019 challenge proposals, organizers had to instantiate at least 90% of a reduced 40-parameter set; among 25 submissions, the median share instantiated was 100%, with a minimum of 94%.Source: §II · p. 3
  • The checklist was finalized after parameter revisions, independent co-author ratings and feedback, and resolution of remaining conflicts.The final list contains 42 main parameters and 79 sub-parameters.Source: §II · p. 3
  • The recommendations address how to report the design and outcomes of individual challenge tasks, with a reviewer checklist provided in Appendix A and Zenodo.Source: §II · p. 3

III. GUIDELINE FOR CHALLENGE REPORTING

The guideline recommends a standard challenge-paper structure and summarizes the challenge mission and outcome in front matter.

III. GUIDELINE FOR CHALLENGE REPORTING

  • Challenge papers are recommended to follow Introduction, Methods, Results, and Discussion, with Methods describing design and Results describing outcome.Source: §III · p. 3
  • A concise title and abstract should precede the main sections, followed by representative keywords summarizing the challenge mission and outcome.Source: §III · p. 3
  • Appendix A provides a structured summary of the section’s recommendations for writing the individual sections.Source: §III · p. 3

A. Title, Abstract and Keywords

The title, abstract, and keywords should give readers a concise account of the challenge’s mission, design, and results.

A. Title, Abstract and Keywords

B. Introduction: Research Context

The introduction recommends that challenge papers explain their biomedical and technical motivation and make the task and its intended contribution clear.

C. Methods: Reporting of challenge design

The recommendations cover challenge organization and policies, the mission and cohorts, datasets and their quality, assessment design, and statistical and further analyses.

Challenge organization

Mission of the challenge

Assessment method

Challenge data sets

Assessment method

D. Results: Reporting of Challenge Outcome

The outcome section recommends reporting submissions and selected teams, metric values, rankings and robustness analyses, and any further analyses.

Challenge submission and participating teams

Metric values

Rankings

Further analyses

E. Discussion: Putting the Results into Context

The BIAS checklist aims to make challenge reporting more transparent and to standardize review, with the goal of improving interpretability and reproducibility. The discussion describes its scope, possible extensions, and use in challenge registration.

E. Discussion: Putting the Results into Context

  • The BIAS checklist is intended to standardize challenge-paper review and improve the interpretability and reproducibility of reported results.The authors present it as a guideline for reporting challenge design and results.Source: E. Discussion: Putting the Results… · p. 6
  • The checklist relies heavily on a previously published challenge-parameter list, and may need updating as probabilistic outputs become more common.The authors say it is currently unknown how such outputs, which can provide a range of plausible solutions, may be efficiently handled in future challenge design.Source: E. Discussion: Putting the Results… · p. 6
  • The guideline was developed for reporting one challenge task; for overlapping tasks in one paper, authors can separate checklists or distinguish shared from task-specific items.The authors prefer separate publications when a challenge includes multiple tasks.Source: E. Discussion: Putting the Results… · p. 6
  • The checklist does not explicitly address collaborative challenges in which teams jointly solve a problem rather than benchmark different methods.The authors describe collaborative challenges as an alternative to competition-focused benchmarking.Source: E. Discussion: Putting the Results… · pp. 6–7
  • Challenge registration extended the reporting work to design review before execution, and the checklist and design document could be adapted to other research areas.For MICCAI 2020, accepted challenges had to post their complete designs before execution; 8 of 28 later reported design changes, most involving schedules attributed to the COVID-19 outbreak.Source: E. Discussion: Putting the Results… · p. 7

COMPETING INTERESTS

  • Anne Martel is a co-founder and CSO of Pathcore; the remaining authors declare no competing interests.Source: COMPETING INTERESTS · p. 7

APPENDIX

The appendix checklist prompts authors to make essential challenge information explicit in the title, abstract, and keywords.

APPENDIX

  • Titles should convey essential challenge information, including that the work concerns biomedical image analysis and its image modality and task.The checklist recommends commonly used terms for modality and task.Source: APPENDIX · p. 9
  • The abstract should summarize the challenge purpose, design, and results, and report its main conclusions.Source: APPENDIX · p. 9
  • Keywords should list the primary terms that characterize the challenge.Source: APPENDIX · p. 9

INTRODUCTION

These recommendations promote transparent reporting of biomedical image analysis challenges by specifying how to document challenge definition, organization, policies, submissions, oversight, and data context. They also call for reporting algorithms’ targets and optimization priorities, the metrics and rankings used to assess them, and the devices used to acquire challenge data.

INTRODUCTION

  • Challenge reporting should explain its biomedical and technical motivation, technical state of the art, and envisioned short- or long-term impact.The checklist also calls for a concise statement of the challenge objective and task.Source: INTRODUCTION · p. 9
  • Authors should identify the challenge objective and task, give a representative name and acronym, and describe the organizing team.These items make the challenge and its organizers explicit.Source: INTRODUCTION · p. 9
  • Reports should state whether the challenge is one-time or recurring and identify its venue, operating platform, and website when available.The checklist asks authors to explain continuation after the study and report the associated event and platform.Source: INTRODUCTION · p. 10
  • Organizers should specify permitted algorithm interaction, training-data rules, and participation eligibility for members of organizers’ institutes.The checklist gives examples of limiting methods to semi-automatic approaches and restricting training data to challenge-provided or public data.Source: INTRODUCTION · p. 10
  • Reports should define award, results-announcement, and publication policies, including how team results and authorship are handled.The checklist also raises separate publication-policy questions about independent team publications and embargo periods.Source: INTRODUCTION · p. 10
  • Authors should document result-submission methods, opportunities to evaluate algorithms before final submission, and a timetable of challenge milestones.The timetable may include data releases, registration, workshops, and results announcements.Source: INTRODUCTION · p. 11
  • Reports should disclose ethics approval, data-use and distribution licenses, code accessibility, conflicts of interest, funding, test-case access, and author contributions.The checklist asks for explicit licensing and access information for both organizers’ evaluation software and participating teams’ code.Source: INTRODUCTION · pp. 11–12
  • The challenge description should identify its application field and task category, distinguish target from challenge cohorts, and report imaging and associated data characteristics.Requested context includes patient or acquisition information and the target entity or anatomical origin of images.Source: INTRODUCTION · pp. 12–13
  • Specify the structure, subject, object, or component that participating algorithms target, and distinguish the target from the challenge cohort when necessary.Examples include a brain tumor, medical-instrument tip, operating-theater nurse, or catheter in a fluoroscopy scan.Source: INTRODUCTION · p. 13
  • Identify which algorithm properties should be optimized, reflect them in the metrics, and prioritize them when combining metrics to rank submissions.When multiple properties are assessed, their priorities should be stated if appropriate.Source: INTRODUCTION · pp. 13–14
  • Specify the devices used to acquire the challenge data.Source: METHODS Challenge data sets · p. 14

METHODS Challenge data sets

The challenge-data and assessment checklist asks authors to document how cases and annotations are assembled, how algorithms are assessed, and how participation and results are reported.

METHODS Challenge data sets

  • Report imaging and data-source details, acquisition centers, and relevant characteristics of people or objects involved in data acquisition.The checklist also asks authors to explain when acquisition information cannot be provided.Source: METHODS Challenge data sets · p. 14
  • Define a case and report total and training, validation, and test counts, explaining the split and other important case characteristics.Examples include class distributions and the rationale for choosing them.Source: METHODS Challenge data sets · pp. 14–15
  • Describe how reference annotations were produced, including annotator instructions and expertise, and how multiple annotations were merged.Provide information separately for training, validation, and test cases when necessary.Source: METHODS Challenge data sets · p. 15
  • Describe preprocessing of raw training data and relevant error sources, estimating their magnitude where possible.The checklist suggests inter- and intra-annotator variability as examples for estimating annotation error.Source: METHODS Challenge data sets · pp. 15–16
  • Define and justify assessment metrics, explain how case-level results are aggregated into rankings, and report statistical methods and their rationale.The checklist also asks how missing test-case results are handled.Source: METHODS Challenge data sets · p. 16
  • Report challenge participation, including registrations and the teams that provided valid submissions.Source: METHODS Challenge data sets · p. 16, Challenge outcome · p. 17

Challenge outcome

The BIAS checklist recommends transparent reporting of challenge purpose, organization, data, assessment methods, metrics, rankings, and follow-up analyses. It calls for discussion of their biomedical and technical significance, limitations, and implications to support interpretation and reproducibility of challenge results.

Challenge outcome

  • Report raw or aggregated metric values, variability, participating teams, and test-set submission counts for every metric.The checklist also requests method descriptions and information about selected participating teams.Source: Challenge outcome · p. 17
  • Report team rankings, the number of test-set submissions per team, and results of statistical analyses.Source: Challenge outcome · p. 17
  • Present further analyses, such as ensembling, inter-algorithm variability, and common problems or biases, and summarize the challenge’s main results.Source: Challenge outcome · p. 17, DISCUSSION · p. 17

DISCUSSION

  • Describe the challenge’s expected biomedical and technical impact in relation to the state of the art and its motivating aims.Source: DISCUSSION · p. 17
  • Assess whether the task is satisfactorily solved, analyze cases where most algorithms perform poorly, and discuss method advantages, disadvantages, and resource use.Source: DISCUSSION · p. 17
  • Discuss challenge-design limitations, provide future-work and maintenance recommendations, and conclude concisely from the study’s results.Source: DISCUSSION · p. 18

SUMMARY

  • The summary should identify the challenge, explain its biomedical and technical purpose and intended impact, and list its defining keywords.A short acronym is preferable when one exists.Source: SUMMARY · p. 25

CHALLENGE ORGANIZATION

  • Report organizers and contacts, the challenge lifecycle, associated event, platform and website, and the schedule from data release through results.Specify whether the challenge is a one-time event, an open call, or a repeated event, and whether it will continue afterward.Source: CHALLENGE ORGANIZATION · pp. 25–27
  • State participation, award, result-announcement, publication, and submission policies, including pre-final evaluation opportunities and access to evaluation and team code.Also report ethics approval, data-use licenses, conflicts and funding, test-label access, and relevant submission instructions.Source: CHALLENGE ORGANIZATION · pp. 26–27

MISSION OF THE CHALLENGE

  • Describe the application field, task category, target and challenge cohorts, imaging modality, context information, algorithm target, and prioritized assessment aims.The checklist distinguishes the intended final-application population or objects from those represented in the challenge data.Source: MISSION OF THE CHALLENGE · pp. 28–30

CHALLENGE DATA SETS

  • Document data sources and acquisition, define a case, report training, validation, and test-set sizes and characteristics, and justify the case and split choices.Acquisition reporting includes devices, imaging protocols, centers or sources, and relevant characteristics of people or objects involved.Source: CHALLENGE DATA SETS · pp. 30–31
  • Describe reference-annotation methods, annotator instructions and expertise, how multiple annotations are merged, preprocessing, and relevant error sources with estimated magnitudes when possible.Provide these details separately for training, validation, and test cases when necessary.Source: CHALLENGE DATA SETS · pp. 31–32

ASSESSMENT METHODS

  • Define and justify metrics that reflect the assessment aims, identify those used for rankings, and explain ranking aggregation, missing-result handling, and the chosen ranking scheme.The checklist calls for a description of how case-level metric results yield final scores or ranks.Source: ASSESSMENT METHODS · p. 32
  • Report and justify statistical methods, including missing-data handling and ranking variability, and describe applicable further analyses such as ensembling or inter-algorithm variability.Further analyses may also examine common problems or biases in submitted methods and ranking variability.Source: ASSESSMENT METHODS · p. 33

Related papers

Loading 1910.04071v5…