Source-linked AI summary
Deep reinforcement learning in medical imaging: A literature review
S. Kevin Zhou, Hoang Ngan Le, Khoa Luu, Hien V. Nguyen, Nicholas Ayache
TL;DR
The paper addresses the incomplete systematic understanding of DRL’s strengths in medical imaging by reviewing its methods and applications. It reports DRL as an effective alternative for difficult optimization problems, while noting challenges remain before its full potential is realized.
Problem
DRL’s application to medical imaging lacks a fully developed systematic understanding of its strengths.
Method
The paper provides a comprehensive tutorial of DRL and reviews applications across medical imaging analysis, optimization, and miscellaneous tasks.
Results
DRL has been demonstrated to be an effective alternative for difficult optimization problems, including parameter tuning, augmentation selection, and neural architecture search.
Takeaways & Limitations
DRL offers a broad approach for addressing difficult optimization problems in medical imaging.
Takeaways & Limitations
Realizing DRL’s full potential for medical imaging requires solving several challenges.
Abstract
from arXiv · showhide
Deep reinforcement learning (DRL) augments the reinforcement learning framework, which learns a sequence of actions that maximizes the expected reward, with the representative power of deep neural networks. Recent works have demonstrated the great potential of DRL in medicine and healthcare. This paper presents a literature review of DRL in medical imaging. We start with a comprehensive tutorial of DRL, including the latest model-free and model-based algorithms. We then cover existing DRL applications for medical imaging, which are roughly divided into three main categories: (I) parametric medical image analysis tasks including landmark detection, object/lesion detection, registration, and view plane localization; (ii) solving optimization tasks including hyperparameter tuning, selecting augmentation strategies, and neural architecture search; and (iii) miscellaneous applications including surgical gesture segmentation, personalized mobile health intervention, and computational model personalization. The paper concludes with discussions of future perspectives.
1. Introduction
Deep reinforcement learning combines reinforcement learning with deep learning to handle sequential, delayed, and non-differentiable medical-imaging problems. The paper addresses limited systematic understanding by providing an expanded tutorial and literature review of DRL methods and applications.
- DRL learns action sequences that maximize expected reward while leveraging deep neural networks’ representative power.
- Unlike supervised learning, DRL supports sequential decisions and highly delayed supervision, including outcomes available only after multiple time steps.
- DRL can optimize non-differentiable objectives such as classification accuracy through network-architecture or parameter-setting search.
- In medical imaging, DRL can search efficient detection sequences, transformation sequences for registration, and region-by-region processing strategies that reduce memory footprint.
- Medical-imaging applications remain incompletely explored partly because DRL’s strengths and weaknesses on medical data lack systematic understanding.
- The paper expands a MICCAI 2018 tutorial into a comprehensive introduction to DRL theories, practice, future directions, and medical-imaging applications.
2. Basics of Reinforcement Learning
Reinforcement learning models sequential decision-making as agent–environment interaction, using rewards to optimize behavior under Markovian dynamics. The review introduces MDPs, value and policy methods, and model-based versus model-free approaches.
- RL learns an optimal policy through trial and error for sequential decision-making, receiving scalar rewards rather than supervision.
- Markov Decision Process: An MDP specifies states, actions, transition probabilities, rewards, and a discount factor for modeling agent–environment decisions.
- Markov Decision Process: At each step, the agent selects an action, receives a reward, and transitions to a new state until termination or a maximum time step.
- Value functions: Return is the discounted weighted sum of rewards across an episode, and value functions estimate expected return under a policy.
- RL categories: Model-based methods use transition and reward dynamics, whereas model-free methods estimate these functions from experience; policy-gradient and value-based algorithms are common model-free methods.
- RL methods: Policy-search methods optimize policy parameters directly, while value-based methods learn value or Q-functions to select actions.
3. Introduction to Deep Reinforcement Learning
Deep reinforcement learning combines reinforcement learning with deep learning, representing policies and value functions with neural networks. This representation supports continuous states or actions and includes both model-free and model-based algorithms.
- DRL combines RL and DL, expressing policies and value functions with neural networks.
- Neural-network representations allow DRL to handle continuous states or actions that are difficult to represent with tables.
- DRL is categorized into model-free and model-based algorithms, which the section reviews separately.
3.1. Deep Learning: Review
The review introduces core deep-learning architectures used in DRL, including autoencoders, deep belief networks, CNNs, and RNNs. These models support representation learning, image processing, and sequence modeling, with RNNs facing long-term dependency difficulties.
- The reviewed deep-learning algorithms include autoencoders, deep belief networks, convolutional neural networks, and recurrent neural networks.
- Autoencoder: Autoencoders support unsupervised representation learning for feature selection or dimensionality reduction, while VAEs learn parametric latent-variable models.
- Deep belief network: Deep belief networks and deep autoencoders can initialize networks instead of random initialization and learn higher-order feature correlations.
- Convolutional neural network: CNNs use shared weights, spatial pooling, and convolutional filters to learn shift-invariant image features while reducing sensitivity to small shifts and distortions.
- Recurrent neural network: RNNs process sequences through hidden states whose outputs depend on previous computations, while LSTMs and GRUs address dependency modeling with gating mechanisms.
- Recurrent neural network: Training RNNs to capture long-term dependencies is difficult, motivating recurrent architectures with memory cells or adaptive gating.
3.2. Model-free DRL algorithms
Model-free DRL algorithms are organized into value-based, policy-gradient, and actor-critic approaches. The section reviews representative methods and training trade-offs, including DQN improvements, recurrent extensions, variance reduction, and asynchronous versus synchronous updates.
- Model-free DRL uses value-based, policy-gradient, and actor-critic approaches.
- Value-based DRL methods: Dueling DQN separates value and advantage representations to address limitations in standard Q-value updating.Its computational graph explicitly approximates V* and introduces an advantage function.
- Value-based DRL methods: DRQN incorporates an RNN into DQN to address memory limitations and imperfect information at each decision point.It replaces DQN’s first fully connected layer with an RNN.
- Policy gradient DRL methods: Policy gradients optimize policies directly, but naive REINFORCE is time-consuming to converge and can encounter local optima.Variance-reduction techniques add terms that reduce variance without affecting the expectation.
- Value-based DRL methods: DQN estimates Q-values from high-dimensional inputs, while Double DQN reduces overestimation caused by the max operation.DQN uses deep neural networks and CNNs for high-dimensional state spaces; Double DQN combines double Q-learning with DQN.
- Actor-critic DRL algorithm: Actor-critic methods combine an actor that selects actions with a critic that evaluates them using value functions.A3C executes agents asynchronously for memory efficiency, whereas A2C synchronously averages worker gradients before updating the global network.
3.3. Model-based algorithms
Model-based DRL learns environmental dynamics to simulate transitions and plan without direct interaction. The section covers planning, value-targeted regression, and policy-search methods, while noting benefits from prior knowledge and challenges with high-dimensional pixels and model bias.
- Model-based DRL learns a transition model that enables environment simulation without direct interaction.The review distinguishes it from model-free approaches, which learn from experience through function approximation.
- Prior knowledge can improve convergence, trained-model quality, and training speed while reducing required training samples.
- Model-based DRL has difficulty using raw pixel data because pixel observations are high-dimensional.DRL addresses this by embedding observations into a lower-dimensional space with autoencoders.
- Model-based DRL approaches: Model-based DRL is categorized into planning on given transitions, planning on learned transitions, and end-to-end learning of planning and transitions.
- Planning methods: DNN-based transition models can predict future frames and rewards, after which Monte Carlo tree search plans action sequences.One described model uses the last four first-person-view frames and the current action as input.
- Policy search methods: MB-MPO learns an ensemble of dynamics models and adapts one policy to each model with one policy-gradient step, exhibiting less model bias.The reported result does not require conservative behavior.
3.4. Useful techniques to train an agent
The review describes practical techniques for stabilizing and improving DRL training, including experience replay, minibatch learning, target-network freezing, and reward clipping.
- Experience replay reuses selected transitions during off-policy learning, including rare but important experiences.It also removes correlations in training data and reduces oscillation of the learning procedure.
- Minibatch learning processes multiple training samples per step and improves robustness to outliers and noise.
- Target Q-network freezing uses separate interacting and target networks, with target weights fixed and slowly updated.The interacting network generates target Q-values used to calculate losses.
- Reward clipping constrains rewards to the range (-1,1) to keep their scale reasonable and support learning.
4. DRL in Medical Imaging
The review surveys DRL applications in medical imaging across parametric analysis, non-differentiable optimization, and miscellaneous topics. The cited literature indicates growing publication activity, while the reference list is explicitly non-exhaustive.
- Parametric medical image analysis: Medical-imaging DRL applications include parametric analysis tasks such as landmark detection, image registration, and view-plane localization.The review organizes references by task, image modality, anatomy, and additional remarks.
- Optimization tasks: DRL is also applied to hyperparameter tuning, image-augmentation selection, and neural architecture search as non-differentiable optimization tasks.Exhaustive grid search for these tasks is described as time-consuming.
- Miscellaneous applications: Miscellaneous applications use DRL for topics beyond parametric analysis and optimization.
- 49 references are summarized across the medical-imaging application categories, but the list is explicitly not exhaustive.Most listed papers were published in journals such as IEEE Transactions on Medical Imaging and Medical Image Analysis or at MICCAI.
- The number of DRL papers in medical imaging published each year shows a growing trend.This trend is illustrated in Fig. 8.
4.1. DRL for parametric medical image analysis
This section formulates parametric medical image analysis as DRL over model parameters, defining actions, states, and rewards for iterative search. Applications include landmark detection, registration, and other localization tasks, with reported gains in accuracy, robustness, and efficiency.
- General formulation: Parametric analysis estimates model parameters θ from an image I, and common tasks use relatively low-dimensional parameterizations.The associated parameters include translation x, y, z, rotation α, β, γ, and scaling s.
- General formulation: The DRL formulation specifies an action space, state, and reward before applying DQL to learn actions that maximize the Q-function.States describe the environment and agent after actions; rewards encourage reaching or approaching the target and penalize otherwise.
- General formulation: Reachability is guaranteed up to quantization error by accumulating parameter-changing actions from an initial guess toward an arbitrary target.One action definition changes one parameter while keeping the others fixed, producing an action space with |A| = 2n.
- Landmark detection: For 3D landmark detection, a multi-scale artificial agent moves in six one-voxel directions and searches from coarse global context to fine local context.The coarse-to-fine process continues across scales until convergence at the finest scale.
4.2. Solving optimization using DRL
This section reviews DRL for medical-image optimization, including hyperparameter and augmentation selection, neural architecture search, and adaptive acquisition. Reported studies improve classification, segmentation, reconstruction, or efficiency relative to specified baselines.
- Optimization applications: DRL is used for optimization problems involving radiotherapy hyperparameters, image augmentation selection, neural architecture search, and learned image acquisition.The motivation is that DRL can handle non-differential metrics where conventional methods may fail.
- Attention and classification: A jointly trained segmentation agent and classifier use hard attention masks and classification accuracy rewards, with reported gains on MURA and hip-fracture datasets.The classifier receives both original and masked data as inputs.
- Synthetic sample selection: A PPO agent selects synthetic images with reliable and informative features, using classification accuracy as reward to address unrealistic generated samples.Performance improved by 8.1% on cervical histopathology images and 2.3% on lymph-node histopathology images.
- Multi-modal classification: An RL framework automatically learns modality weights for four ultrasound streams, and multi-modal methods outperform single-modal methods on breast-nodule classification.The network uses four stream losses and an additional fusion loss, all weighted through RL.
- Segmentation optimization: RL-based training-strategy search boosts baseline performance, while searched U-Net architectures outperform nnU-Net on Medical Segmentation Decathlon challenges.The architecture search varies input size, pooling, filters, stride, activation, skip connections, and dilation.
- Acquisition optimization: Adaptive acquisition policies reduce metal artifacts and can produce reconstruction quality similar to human expert tuning, with one policy reporting a quality score 10.7% higher than human planners.The scanning policy adaptively adjusts acquisition angles and dose.
4.3. Miscellaneous topics
The reviewed miscellaneous applications extend DRL beyond direct image analysis to video summarization, surgical gesture segmentation, mobile-health intervention, and computational model personalization. These studies report competitive performance, reduced interaction, or gains in robustness and speed.
- Video summarization: DRL video summarization selects frames through sequential decisions based on visual representations and temporal dependencies, achieving state-of-the-art summarization performance with the highest F1 score.The framework combines an encoder-decoder CNN, Bi-LSTM, and RL frame-selection network.
- Surgical gesture segmentation: For surgical gesture segmentation, DRL uses TCN features and variable time steps, achieving performance similar to TCN on JIGSAWS.Large time steps contribute a higher edit score, while small steps help near gesture boundaries.
- Mobile health: Group-driven RL clusters users by similarity so each group learns an RL model, addressing settings where users may resemble some but not all others.The approach targets personalized mobile-health interventions delivered through smart devices.
- Computational model personalization: For computational model personalization, an agent explores parameter effects offline and learns an online personalization strategy for complex physiological models.Applications include cardiac electrophysiology inverse problems and whole-body circulation-model personalization.
- Computational model personalization: The personalization algorithm matches standard methods while achieving up to 11% higher success rates and operating up to seven times faster.These results were reported for cardiac electrophysiology and whole-body circulation model applications.
5. Conclusions and Future Perspectives
DRL has been applied to medical image analysis and optimization, but broader medical-imaging deployment remains constrained by data, reward design, environmental assumptions, user interaction, and reproducibility challenges.
- DRL has been applied to landmark localization, object detection, registration, and other medical image analysis tasks.
- DRL has also been used for parameter tuning, augmentation-strategy selection, and neural architecture search.
- Challenges ahead: Reward functions are difficult to define, especially when intermediate rewards are unavailable and assigning rewards for each action requires careful manual effort.
- Challenges ahead: Medical imaging applications require large training datasets or expert demonstrations, which are expensive and difficult to obtain.
- Challenges ahead: Reviewed approaches generally assume stationary environments, while real scanning applications still require validation with data from real scanning scenarios rather than simulations alone.
- Challenges ahead: Further challenges include limited user interaction, delayed rewards, high-dimensional action spaces, and difficulty reproducing DRL results because of nondeterministic factors.
Credit authorship contribution statement
The credit authorship statement assigns conceptualization and writing responsibilities across the listed contributors, with review and editing shared among several authors.
- S. Kevin Zhou contributed conceptualization, original drafting, and review and editing.
- Hoang Ngan Le and Khoa Luu contributed conceptualization, original drafting, and review and editing.
- Hien V. Nguyen contributed conceptualization, original drafting, and review and editing, while Nicholas Ayache contributed review and editing.