PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Reinforcement Learning in Computer Vision: A Comprehensive Survey

Ngan Le, Vidhiwar Singh Rathour, Kashu Yamazaki, Khoa Luu, Marios Savvides

arXiv:2108.11510v1cs.CVcs.AI

TL;DR

Computer-vision research needs a consolidated account of how deep reinforcement learning methods are used across diverse decision-making tasks. This paper surveys RL, DL, and DRL foundations, organizes applications across seven computer-vision categories, and reviews techniques, architectures, performance, datasets, and code availability; reported examples include up to 25% accuracy improvement in 3D MRI registration. The survey also identifies reward-design and environment-complexity challenges and discusses future directions.

  • Problem

    Computer-vision research needs consolidated coverage of deep reinforcement learning foundations and applications across diverse decision-making tasks.

  • Method

    The paper synthesizes deep-learning, reinforcement-learning, and deep-reinforcement-learning foundations, then categorizes computer-vision applications and examines techniques, network design, performance, datasets, and source-code availability.

  • Results

    Up to 25% accuracy improvement was obtained on 3D MRI image registration using reinforcement learning with uncertainty evaluation.

  • Takeaways & Limitations

    The survey provides a structured reference for understanding DRL applications across seven computer-vision areas and their reported methods and performance.

  • Takeaways & Limitations

    Reward functions are difficult to define, and reviewed approaches generally assume stationary environments although real systems are often partially observable and non-stationary.

Abstract

from arXiv · show

Deep reinforcement learning augments the reinforcement learning framework and utilizes the powerful representation of deep neural networks. Recent works have demonstrated the remarkable successes of deep reinforcement learning in various domains including finance, medicine, healthcare, video games, robotics, and computer vision. In this work, we provide a detailed review of recent and state-of-the-art research advances of deep reinforcement learning in computer vision. We start with comprehending the theories of deep learning, reinforcement learning, and deep reinforcement learning. We then propose a categorization of deep reinforcement learning methodologies and discuss their advantages and limitations. In particular, we divide deep reinforcement learning into seven main categories according to their applications in computer vision, i.e. (i)landmark localization (ii) object detection; (iii) object tracking; (iv) registration on both 2D image and 3D image volumetric data (v) image segmentation; (vi) videos analysis; and (vii) other applications. Each of these categories is further analyzed with reinforcement learning techniques, network design, and performance. Moreover, we provide a comprehensive analysis of the existing publicly available datasets and examine source code availability. Finally, we present some open issues and discuss future research directions on deep reinforcement learning in computer vision

1 Introduction

Deep reinforcement learning combines reinforcement learning with deep neural-network representations for sequential decision-making, and this survey reviews its computer-vision applications. It covers foundational theories, application categories, datasets, code availability, challenges, and future directions.

  • Motivation: Deep reinforcement learning combines reinforcement learning and deep learning to address complex sequential decision-making problems through interaction and trial-and-error reward maximization.Unlike supervised learning, it can sample training data from an environment rather than requiring large labeled datasets.
  • Scope and objectives: The survey aims to explain RL and DRL principles while comprehensively reviewing recent DRL applications for computer vision.It positions this coverage alongside existing surveys focused on algorithms, models, generalization, practical applications, and healthcare.
  • Organization: The paper organizes computer-vision applications across landmark detection, object detection, object tracking, image registration, image segmentation, video analysis, and other applications.The application sections analyze reinforcement-learning techniques, network design, and performance.
  • Organization: The survey introduces deep-learning, reinforcement-learning, and deep-reinforcement-learning fundamentals before reviewing model-based, model-free, value-based, policy-gradient, and actor-critic techniques.It begins with MLPs, autoencoders, CNNs, and RNNs, then develops RL from MDPs through value and Q-functions.
  • Future directions: The paper concludes with future perspectives addressing challenges of DRL in computer vision and recent advanced techniques.These perspectives are presented in the final section after the application-focused review.

2 Introduction to Deep Learning

Deep learning models use layered neural networks to transform inputs into outputs, while specialized architectures support representation learning, spatial processing, and sequence modeling. This section introduces perceptrons and MLPs, unsupervised models, CNNs, and RNNs, including their core operations and training considerations.

  • Perceptron and MLP: A perceptron is a computational model with one or more inputs, a processor, and one output; a neuron computes o = f(x, θ) using weights and a bias.The activation function σ is point-wise and may be Tanh, sigmoid, softmax, ReLU, or LeakyReLU.
  • Perceptron and MLP: An MLP contains input and output layers separated by one or more hidden layers, with each layer potentially containing many units.The example described has 3 input units, 3 hidden units, and 1 output unit.
  • Unsupervised architectures: Autoencoders support unsupervised representation learning, including feature selection and dimension reduction, while deep autoencoders and DBNs can initialize networks.DBNs are based on restricted Boltzmann machines whose hidden units learn features capturing high-order data correlations.
  • Convolutional Neural Networks: CNNs use weight sharing, spatial correlation, and pooling to learn shift-invariant image features and reduce sensitivity to small shifts and distortions.A typical CNN applies convolution and optional pooling across stages, followed by point-wise nonlinear activation; its outputs are feature maps.
  • Convolutional Neural Networks: CNNs have achieved state-of-the-art performance in image understanding, video analysis, and audio or speech recognition, with applications including tracking, segmentation, pose estimation, and action recognition.The survey describes CNNs as useful across multiple computer-vision tasks beyond image classification.
  • Recurrent Neural Networks: RNNs process sequential inputs using hidden states whose outputs depend on previous computations, and they are trained using backpropagation through time and gradient descent.LSTM and GRU variants address the difficulty of capturing long-term dependencies by adapting memory or dependency timescales.

3 Basics of Reinforcement Learning

Reinforcement learning models sequential decision-making as an agent interacting with an environment to maximize cumulative reward. The section formalizes this process with MDPs, policies, value functions, Q-functions, and model-based or model-free methods.

  • RL agents learn optimal policies through trial-and-error interaction with environments, receiving scalar feedback that may be delayed.
  • Markov Decision Process: An MDP represents RL using states, actions, transition probabilities, rewards, and a discount factor.
  • Agent–Environment Interaction: A policy maps states to actions, while roll-outs generate trajectories and finite trajectories are treated as episodes.
  • Objective: The RL objective is to find an optimal policy that maximizes discounted expected cumulative reward, or return.
  • Value and Q-functions: Value functions evaluate states under a policy, whereas Q-functions evaluate actions in states and provide action-specific information when value functions alone cannot reconstruct an optimal policy.
  • RL Method Categories: RL methods include value-function, transition-model, policy-search, and return-function approaches; model-based methods use learned transition and reward functions, which are rarely known in practice.

4 Introduction to Deep Reinforcement Learning

Deep reinforcement learning combines reinforcement learning with deep learning, representing values and policies with neural networks. This enables DRL to handle continuous states or actions and supports model-based and model-free algorithm categories.

  • DRL combines RL and DL, using neural networks to represent value functions and policies.
  • Neural-network representations allow DRL to handle continuous states or actions that are difficult to represent with tables.
  • DRL algorithms are categorized into model-based and model-free methods, paralleling the corresponding RL categories.

4.1 Model-Free Algorithms

Model-free DRL methods learn value functions or policies without an explicit environment model. The section covers value-based, policy-gradient, and actor-critic algorithms, along with representative DQN variants and training procedures.

  • Model-free DRL implementations use value-based or policy-gradient methods.
  • Value-based DRL methods: DQN uses CNNs to map raw pixels to Q-values estimating future rewards for all actions.
  • Value-based DRL methods: Double DQN reduces overestimation by separating action selection from action evaluation.
  • Value-based DRL methods: Dueling DQN separately estimates state value and action advantage before combining them into Q-values.
  • Value-based DRL methods: DRQN replaces DQN’s first fully connected layer with an RNN to address limited memory and imperfect information.
  • Policy gradient DRL methods: Policy-gradient methods optimize policies directly, while REINFORCE estimates gradients with Monte Carlo sampling but can converge slowly and reach local optima.
  • Actor-Critic DRL algorithm: Actor-critic methods combine policy-gradient and value-based ideas by using a critic to estimate expected returns for policy-gradient computation.
  • Actor-Critic DRL algorithm: A3C avoids experience replay by asynchronously running agents in parallel and updating a shared global network.

4.2 Model-Based Algorithms

Model-based DRL learns environment dynamics through transition models, enabling simulation and planning without direct interaction at every step. The survey covers value-function, policy-search, and on-policy/off-policy classifications.

  • Model-based DRL learns transition dynamics from experience to simulate the environment without direct interaction.
  • Value function: DNN-based transition models can predict next frames and rewards for Monte Carlo tree search planning.
  • Policy search: Policy-search methods directly find policies using gradient-free or gradient-based optimization, including trust-region and model-based meta-policy approaches.
  • The survey summarizes model-based and model-free algorithms and categorizes them as on-policy or off-policy.

4.3 Good practices

Experience replay improves off-policy DRL training by reusing past interactions and reducing correlations in training data.

  • Experience replay removes correlations from past experiences, reduces learning oscillation, and improves data efficiency through reuse.

5 DRL in Landmark Detection

DRL frames anatomical landmark localization as sequential decision-making, allowing agents to move image regions toward target landmarks. Surveyed methods use multiscale search, DQN, actor-critic, and multiple-agent strategies.

  • Manual landmark annotation is time-consuming, tedious, and prone to errors, motivating automated detection.
  • Anatomical landmark detection can be formulated as an MDP in which agents learn search behavior from image information.
  • Single Landmark Detection: Multiscale search moves from coarse to fine image scales, using distance-based rewards to guide agents toward landmarks.Experiments on 3D CT scans reported a 20-30% average accuracy increase and lower distance error than SADNN and 3D-DL.
  • Single Landmark Detection: DQN-based fetal-ultrasound localization uses six directional actions and rewards changes in Euclidean distance to the target.
  • Single Landmark Detection: Actor-critic localization observes surrounding voxel blocks and searches using the same six directional movements.
  • Multiple Landmark Detection: Some methods use partial policies or shared knowledge because anatomical landmarks are interdependent and may require heterogeneous policies.
  • Multiple Landmark Detection: Multiple-landmark detection assigns one agent to each landmark and uses clipped distance-difference rewards to improve efficiency and robustness to missing data.
  • A general DRL landmark detector shifts an ROI across the image, with rewards reflecting improvement toward the ground-truth location.

6 DRL in Object Detection

DRL reframes object detection and localization as sequential decision-making, allowing agents to iteratively transform regions or bounding boxes. Across reviewed methods, reported results include improved accuracy, reduced runtime, or fewer region proposals on Pascal VOC and medical imaging datasets.

  • Sequential formulations: DRL object localization models formulate bounding-box refinement as an MDP with discrete translation, scaling, and terminal actions.States commonly include region features and action history, while rewards use changes in IOU or localization quality.
  • Sequential formulations: Hierarchical and tree-structured agents progressively select and refine image regions using DQN-based policies.Tree-RL uses eight translation and five scaling actions to form a hierarchy of localized regions.
  • Reported performance: Sequential visual detection achieved comparable mAP with lower runtime than exhaustive sliding-window, CPMC, and region-proposal searches.The comparison was reported on the Pascal VOC 2012 object detection challenge.
  • Reported performance: Tree-RL with Faster R-CNN outperformed RPN with Fast R-CNN in AP and produced results comparable to Faster R-CNN.These results were reported on Pascal VOC 2007 and 2012.
  • Reported performance: DRL-based active breast lesion detection reported comparable true- and false-positive proportions with supervised-learning and Ms-C methods, but lower mean inference time.The evaluation used DCE-MRI and T1-weighted anatomical data.

7 DRL in Object Tracking

DRL object tracking methods model bounding-box movement across frames as sequential decisions for single or multiple objects. The reviewed approaches use DQN, actor-critic, REINFORCE, and recurrent architectures, with reported gains in tracking accuracy, landmark error, success, precision, or runtime depending on the task.

  • Single-object tracking: Dual-agent deformable face tracking jointly predicts facial bounding boxes and landmarks using separate tracking and alignment agents.The method uses eight actions, including translation, scaling, stop, and continue.
  • Single-object tracking: Single-object trackers use consecutive frames and bounding-box actions to update object location through translation and scale changes.Rewards include thresholded IOU or overlap-based measures, and some methods combine CNNs with LSTMs or actor-critic learning.
  • Multi-object tracking: Multi-object tracking methods formulate object trajectories as MDPs or collaborative Q-networks, with states representing object status, positions, or temporal history.One approach uses Active, Tracked, Lost, and Inactive states; another jointly performs detection and tracking.
  • Single-object tracking: Facial landmark tracking reported lower normalized point-to-point error and higher facial-tracking success than ICCR, MDM, Xiao et al., and other comparisons.The cited method uses an LSTM-based facial tracking design.
  • Single-object tracking: A single-object tracker achieved higher success-overlap and precision-location-error areas under the curve than CREST, ADNet, MDNet, HCFT, SINT, DeepSRDCF, and HDT.Both success and precision comparisons were reported against the listed trackers.
  • Multi-object tracking: Reported multi-object tracking accuracy and precision were comparable with several competing methods, while one recurrent approach incurred higher running time.The metrics cited are MOTA and MOTP.

8 DRL in Image Registration

DRL image-registration methods treat alignment as sequential transformation of a floating or moving image toward a fixed image. The reviewed studies apply value-based, actor-critic, recurrent, and uncertainty-aware methods to 2D or 3D medical registration under deformation and appearance variability.

  • Reported performance: Uncertainty-aware reinforcement learning improved probabilistic registration accuracy by up to 25% on 3D MRI images.The method predicted registration error with regression random forests.
  • Reported performance: DRL registration reported better success rates than ITK, Quasiglobal, and semantic registration methods.The evaluation used abdominal spine and cardiac CT/CBCT datasets.
  • Reported performance: One method achieved lower Euclidean distance error than Hausdorff, ICP, DQN, and Dueling-DQN comparisons on a thorax and abdomen dataset.The model used a 3D convolutional architecture.
  • Registration methods: A multimodal 3D method uses Dueling DQN for value and advantage estimation and Double DQN for network-weight updates.Its state contains cropped 3D tensors from both data modalities, and action history helps address oscillations and closed loops.
  • Registration methods: A robust non-rigid method addresses large deformations and appearance variability by optimizing a spatial transformation with DQN actions.The reward corresponds to changes in distance between ground-truth and predicted transformations.
  • Registration formulation: DRL registration agents observe fixed and floating images, apply transformation actions, and receive rewards based on registration or transformation error.This general pipeline is illustrated for image registration, including volumetric data.

9 DRL in Image Segmentation

DRL segmentation methods formulate mask refinement, seed selection, brush control, or pixel and voxel labeling as sequential actions. Reported studies cover interactive, medical, volumetric, and anomaly segmentation, with improvements in IOU, Dice, Hausdorff distance, or classification metrics across several datasets.

  • Medical segmentation: An iteratively refined multi-agent method targets 3D medical segmentation, where reinforcement learning refines an initial coarse segmentation using image and user-hint information.The method addresses limitations reported for prior 3D medical segmentation work.
  • Reported performance: DRL methods reported better IOU than FCN and iFCN for saliency segmentation on the MSRA10K dataset.The reported action design uses two actions per pixel.
  • Reported performance: Iteratively refined multi-agent segmentation reported better performance than MinCut, DeepIGeoS, and InterCNN on BraTS 2015, MM-WHS, and NCI-ISBI 2013 datasets.The comparison covers multiple 3D medical segmentation datasets.
  • Reported performance: Multi-step medical image segmentation reported higher mean Dice and lower Hausdorff distance than Grab-Cut, PSPNet, FCN, U-Net, and other methods.The method uses actor-critic reinforcement learning with a deterministic policy and controls brush-stroke position and shape.
  • Reported performance: Anomaly segmentation reported superior precision, recall, and F1 on MVTec AD and CrackForest compared with U-Net and an unsupervised baseline, but only recall superiority on NanoTWICE.The method attends to a predicted image patch to reduce imbalance between normal and abnormal regions.
  • Segmentation formulations: DRL segmentation agents use image features with masks, probability maps, hints, or user seeds as states and iteratively modify segmentation outputs.Actions can label pixels, adjust voxel probabilities, select seeds, or control brush position and shape.

10 DRL in Video Analysis

DRL methods model video-analysis tasks as sequential decisions, using task-specific states, actions, rewards, and network designs for segmentation, recognition, and summarization. Reported comparisons show improvements across several video benchmarks and evaluation measures.

  • Video object segmentation: Video object segmentation includes unsupervised, weakly supervised, and semi-supervised settings, with DRL methods assigning pixel labels across frames.The surveyed approaches include cutting agents, motion-oriented unsupervised learning, key-frame scheduling, and multiagent frameworks.
  • Segmentation mechanisms: Image and video segmentation agents use actions such as pixel labeling, translation, scaling, frame categorization, or location-prior placement, with rewards based on IOU or segmentation quality.States may include images, masks, consecutive frames, optical flow, action history, or global video information.
  • Action recognition: DRL methods for action recognition model tasks as MDPs and use action-selection policies over frame or skeleton information.The surveyed surgical gesture and skeleton-based approaches use actor-critic or DQN-style action spaces and report gains on multiple datasets.
  • Video object segmentation: Higher mean region similarity, contour accuracy, and temporal stability were reported for DRL video object segmentation than for several semi-supervised and weakly supervised methods.The comparison includes MSK, ARP, CTN, VPN, SeamSeg, BSVS, VSOF, OSVOS, GVOS, and Spftn.
  • Action recognition: Higher cross-subject and cross-view metrics on NTU+RGBD, and higher accuracy on SYSU-3D and UT-Kinect, were reported than for several competing methods.Comparisons include Dynamic Skeletons, HBRNN-L, Part-aware LSTM, LieNet-3Blocks, and Two-Stream CNN.
  • Video summarization: Video summarization uses frame-selection actions with diversity-representativeness rewards, while reported DRL systems achieve higher Fscore or F1scores than multiple baselines.The surveyed methods use DQN or Duel DQN/Double DQN and evaluate on datasets including TVSum, SumMe, and CoSum.

11 Others Applications

The survey reviews diverse DRL applications beyond the main vision categories, covering robotic manipulation, visual control, medical diagnosis, crowd counting, HDR imaging, autonomous driving, fairness, attention, and creative image generation. Across these tasks, methods define task-specific states, actions, and rewards and report improvements over comparison methods or successful task completion.

  • Robotics: Deformable-object manipulation uses modified DDPG with visual observations, robot state, continuous actions, and sparse completion rewards, achieving up to 90% success.The method was evaluated on cloth-folding and cloth-hanging tasks in PyBullet.
  • Robotics: Visual-perception control combines segmentation, control-policy, and visual-guidance modules to control robotic systems from visual input.The perception module uses pretrained DeepLab and ICNet models.
  • Medical Applications: Medical applications use DRL for sub-pixel neural tracking, appendix localization and diagnosis, and ultrasound-guided robot control.Appendix localization uses an actor-critic framework, while ultrasound guidance uses Double-DQN and Duel-DQN with ResNet18 features.
  • Other Applications: Other applications include autonomous driving, facial-recognition bias mitigation, feature-map attention, and algorithmic painting, with reported challenge wins, higher verification accuracy, improved CNN performance, and better resemblance than SPIRAL.The painting agent controls stroke location, shape, color, and transparency using model-based DRL and DDPG.
  • Image and Video Processing: DRL supports crowd counting and automated exposure bracketing by sequentially selecting counting weights or exposure candidates, with higher PSNR reported for HDR generation.The exposure pipeline uses EBSNet for bracketing selection and MEFNet for HDR image generation.

12 Future Perspectives

Future deployment of DRL in computer vision is constrained by reward design, continuous and high-dimensional spaces, complex real-world environments, and substantial data requirements. The survey highlights advanced directions including inverse DRL, multi-agent DRL, meta-DRL, and imitation learning.

  • Challenge Discussion: Real-world DRL rewards are difficult to specify, intermediate rewards may be unavailable, delayed rewards hinder training, and per-action rewards require manual design.Reward construction often requires domain knowledge that may not be available.
  • Challenge Discussion: Continuous state and action spaces challenge standard RL algorithms, so existing work commonly discretizes them.The survey specifically notes that Q-learning generally handles discrete states and actions.
  • Challenge Discussion: High-dimensional action spaces are difficult for Q-function training, leading existing work to use low-dimensional parameterizations typically below 10 dimensions.An exception uses 15-D and 25-D parameterizations for 2D and 3D registration.
  • Challenge Discussion: Real-world systems are often partially observable, non-stationary, stochastic, and noisy, whereas reviewed approaches generally assume stationary environments.Action delays and sensor or action noise distinguish many real systems from simulated environments.
  • Challenge Discussion: DRL requires large amounts of training data or expert demonstrations, while large annotated datasets are expensive and difficult to obtain.The survey points to realworldrl-suite as an open-source environment suite for studying real-world deployment challenges.
  • Future Directions: Inverse DRL, multi-agent DRL, meta-DRL, and imitation learning are identified as promising directions for addressing reward, interaction, generalization, and demonstration-related issues.Meta-RL is described as enabling learning new skills from small amounts of experience.

Conclusion

The paper presents a comprehensive survey of DRL for computer vision, combining theoretical foundations, algorithmic categories, application coverage, and discussion of datasets, code, open issues, and future directions.

  • Conclusion: The survey covers DL foundations, RL techniques, model-based and model-free DRL, and applications spanning landmark detection, object detection, tracking, registration, segmentation, video analysis, and other vision tasks.It also examines datasets, source-code availability, open issues, and future research directions.
Loading 2108.11510v1…