PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Active Learning and Human-in-the-Loop Deep Learning for Medical Image Analysis

Samuel Budd, Emma C Robinson, Bernhard Kainz

arXiv:1910.02923v2cs.LGcs.CVcs.HCeess.IV

TL;DR

Medical image analysis faces costly and variable annotation, imperfect generalization, safety-critical errors, and limited interpretability despite deep learning’s strong performance. This review surveys active learning and human-in-the-loop approaches across data selection, interactive prediction refinement, interpretation, deployment, and future research. It argues that combining these techniques could support accurate and robust clinical systems while retaining meaningful human input.

  • Problem

    Medical image analysis needs reliable clinical deep learning despite costly annotation, imperfect transfer across domains, safety-critical errors, and limited ways to interpret or correct predictions.

  • Method

    The review synthesizes active learning and human-in-the-loop techniques for medical image analysis, covering informative sample selection, interactive refinement, interpretation, deployment, and future directions.

  • Results

    The review evaluates how human feedback, interpretability methods, practical considerations, and related research can shape deep learning systems for clinical practice.

  • Takeaways & Limitations

    Combining active learning and human-in-the-loop computing is proposed as a direction for developing accurate, robust deep learning applications usable in clinical practice.

  • Takeaways & Limitations

    Actively acquired datasets can be coupled to the selecting model and may produce worse performance for successor models than random sampling.

Abstract

from arXiv · show

Fully automatic deep learning has become the state-of-the-art technique for many tasks including image acquisition, analysis and interpretation, and for the extraction of clinically useful information for computer-aided detection, diagnosis, treatment planning, intervention and therapy. However, the unique challenges posed by medical image analysis suggest that retaining a human end user in any deep learning enabled system will be beneficial. In this review we investigate the role that humans might play in the development and deployment of deep learning enabled diagnostic applications and focus on techniques that will retain a significant input from a human end user. Human-in-the-Loop computing is an area that we see as increasingly important in future research due to the safety-critical nature of working in the medical domain. We evaluate four key areas that we consider vital for deep learning in the clinical practice: (1) Active Learning to choose the best data to annotate for optimal model performance; (2) Interaction with model outputs - using iterative feedback to steer models to optima for a given prediction and offering meaningful ways to interpret and respond to predictions; (3) Practical considerations - developing full scale applications and the key considerations that need to be made before deployment; (4) Future Prospective and Unanswered Questions - knowledge gaps and related research fields that will benefit human-in-the-loop computing as they evolve. We offer our opinions on the most promising directions of research and how various aspects of each area might be unified towards common goals.

1. Introduction

Medical imaging supports many clinical decisions, but domain shifts, costly expert annotation, safety-critical errors, and limited interpretability constrain fully automated deep learning. This review examines how active learning and human interaction can support development, deployment, and practical use of medical image analysis systems.

  • Clinical motivation: Medical imaging informs detection, diagnosis, treatment planning, intervention, and therapy, while shortages of qualified radiologists motivate reliable automated methods.Deep learning has become state of the art for many medical image analysis tasks, but clinical uptake remains limited.
  • Clinical challenges: Medical images are difficult to interpret because they are noisy, modality-dependent, and acquired under varied conditions, creating domain gaps across tasks and sites.Models may require extensive relabelling to regain performance in different workflows.
  • Clinical challenges: Expert annotation is costly and variable because medical images require prior knowledge, producing inter- and intraobserver disagreement about ground truth.Multiple expert annotators may be needed to annotate datasets and reach consensus.
  • Clinical challenges: Clinical deployment requires sufficient training data, correction or override of predictions, and meaningful transparency because small errors can have catastrophic downstream consequences.The review identifies these as three key challenges for deep learning applications in clinical settings.
  • Review scope: The review surveys active learning, feedback-driven prediction refinement, interpretability, practical deployment considerations, and future research directions for human-in-the-loop systems.Active learning selects informative unlabelled samples for annotation, potentially reducing annotation burden while maintaining strong performance.

2. Active Learning

Active Learning selects informative medical-image samples for annotation, combining uncertainty, diversity, representativeness, and generative strategies to reduce labeling while maintaining strong performance. Reviewed studies report state-of-the-art or comparable results across segmentation, diagnosis, and synthesis tasks with substantially reduced annotation.

  • Active Learning can achieve state-of-the-art performance for biomedical image analysis using fewer annotations, reducing development costs for deep-learning systems.
  • 2.2.1. Uncertainty: Uncertainty-based methods rank samples by prediction uncertainty, including least confident, margin, and ensemble-based approaches.Least confident sampling uses the lowest class probability, while margin sampling compares the two most probable labels.
  • 2.2.1. Uncertainty: Bayesian Active Learning by Disagreement uses stochastic Bayesian CNN predictions to select samples with high mutual information about model parameters.BatchBALD extends this idea to jointly score batches and reduce redundant selections within each acquired batch.
  • 2.2.2. Representativeness: Representativeness and diversity criteria complement uncertainty by selecting informative, nonredundant samples from the image distribution.Reviewed examples include density-weighted selection, content distance, geometric uncertainty, and Fisher Information; these methods achieved strong segmentation performance with partial annotation.
  • 2.2.3. Generative Adversarial Networks for Informativeness: Generative approaches use GANs or synthetic data to expand training information from limited annotations and reduce required labeling.Reported studies reached fully supervised benchmark performance with 80% of labels and comparable fully annotated performance with 33% of pixels annotated.

3. The Final Percent: Interactive refinement of model outputs

Human-in-the-loop systems let users iteratively refine model predictions on unseen medical images, addressing imperfect generalization and acceptance criteria. The reviewed approaches primarily demonstrate interactive segmentation, while refinement for classification and regression remains less developed.

  • Interactive refinement: Human users can manually adjust automated predictions on unseen images when outputs fail required acceptance criteria.The refinement process is intended to reduce the time and mental effort required for manual annotation or purely manual correction.
  • Interactive segmentation: Interactive segmentation methods incorporate scribbles, seed points, bounding boxes, or initial predictions to produce progressively refined segmentations.Examples include UI-Net, CRF-CNN, DeepIGeoS, BIFSeg, and interCNN-based pipelines.
  • Interactive segmentation: Interactive annotation can also support 3D segmentation by transforming user-guided 2D slice segmentations into high-accuracy 3D surfaces.The Smart-brush approach uses several guided slices and Hermite radial basis functions to generate a 3D surface, illustrating a route to training-data creation.
  • Interactive segmentation: Reinforcement learning can model iterative voxel-level refinement and use previous prediction uncertainty to capture dependencies among voxels.Multi-agent reinforcement learning treats each voxel as an agent with a shared voxel-level behavior strategy.
  • Open challenges: Refinement of classification and regression predictions is less developed than segmentation refinement and remains an open research area.The review also connects interactive refinement with interpretation, user feedback, and model uncertainty in safety-critical medical applications.

4. Practical Considerations

The review examines practical issues in building and deploying human-in-the-loop systems, including annotation quality, annotation formats, interfaces, and dataset transferability. It highlights ways to reduce annotation burden while identifying limits to generalisability and reuse.

  • Annotation quality: Crowd-annotation methods model annotator expertise and reliability to select capable annotators and mitigate noisy labels.Approaches include learning individual annotator quality, modelling label correlations, and using crowd layers to capture reliability and bias.
  • Annotation quality: Clear instructions and learned models of oracle quality can improve non-expert annotation, but task-specific instructions are not generalisable.The review reports that challenging cases can be presented to stronger oracles, while instructions must be developed for each new annotation task.
  • Annotation formats: Weak annotations such as image-level labels, bounding boxes, ellipses, scribbles, and super-pixel selections can reduce annotation effort and support high-performing models.Weak annotations require less expertise and can be acquired faster, while super-pixels largely reduce user annotation load.
  • Annotation formats: Weakly annotated segmentation approaches can be less generalisable when they depend on an atlas.
  • Multi-task learning: Active learning for multi-task settings remains underexplored, and actively acquired datasets may perform worse for successor models than randomly sampled datasets.The review links this concern to coupling between selection strategies and the model used during acquisition.
  • Interfaces and practical deployment: Interface design is central to usability and application success, while future work should jointly assess oracle expertise, annotation type, and annotation cost.The review notes that interface approaches are diverse and that these economic factors are interdependent.

5. Future Prospective and Unanswered Questions

The review identifies future work around combining human-in-the-loop methods, transfer and semi-supervised learning, continual adaptation, and uncertainty-based interaction. It also highlights unresolved questions about cost, forgetting, bias, interoperability, and evaluation.

  • Related learning paradigms: Semi-supervised learning uses limited labelled data and larger unannotated datasets to improve models, offering a potential route for medical imaging.
  • Annotation efficiency: In a video annotation study, six mouse clicks combined with an AFT framework reduced annotation cost by 80% versus training from scratch and 50% versus random sample selection.
  • Related learning paradigms: Transfer learning can reduce the number of annotated samples needed for new tasks, while pre-trained natural-image networks may further reduce annotation effort.
  • Continual learning: Continual learning requires quantifying performance and robustness over a model’s lifespan, including the effects of catastrophic forgetting and the cost of continued training.Elastic weight consolidation reduced catastrophic forgetting in two MRI segmentation tasks, but substantial room for improvement remained.
  • Bias and evaluation: Active-learning bias may sometimes help overparameterized models with little data, but its effects on end-user adaptation and transferability require further study.The review also notes that error costs are not currently quantifiable for deep-learning medical imaging applications.
  • Unified human-in-the-loop systems: Active learning, iterative refinement, and transfer learning could be combined to reduce annotation effort and domain-shift effects.The proposed unified framework would use model predictions as starting points for refinement and train on data from the prediction distribution.
  • Uncertainty and interpretation: Active-learning selection and human-readable uncertainty measures may not be the same, creating a tension between sample selection and interpretability.
  • Evaluation and integration: Human-in-the-loop research has overlapping goals but substantial gaps remain in understanding relationships between methods and their performance.

6. Conclusions

The conclusion presents human-in-the-loop computing as a unifying perspective for incorporating people throughout medical-image deep-learning workflows. It argues that combining active learning and related methods could support clinical systems that are more accurate, robust, efficient, and usable.

  • Conclusions: The reviewed methods share a focus on involving humans in deep-learning development and clinical use, supporting their treatment as human-in-the-loop computing.
  • Conclusions: The review calls for end-to-end systems that combine active learning and human-in-the-loop methods for clinical practice.
  • Conclusions: Such systems are intended to help users achieve better results faster and with less expertise while freeing expert time for challenging cases.

Supplementary Material

Figure 4 presents a table of features demonstrated by work discussed in the review.

  • Figure 4 is organized as a table.
  • The table records features demonstrated by reviewed work.
  • The figure summarizes features across work discussed in this review.
Loading 1910.02923v2…