PaperlayerView exact revision on arXiv

Source-linked AI summary

Reinforcement Learning in Healthcare: A Survey

Chao Yu, Jiming Liu, Shamim Nemati

arXiv:1908.08796v4cs.LGcs.AI

TL;DR

Healthcare decisions often involve prolonged, sequential procedures with delayed feedback, motivating reinforcement learning as an alternative to one-shot supervised approaches. This survey synthesizes RL foundations, techniques, healthcare applications, challenges, and future directions, reporting successful applications across multiple domains. It also identifies practical boundaries involving continuous actions, policy evaluation, exploration, and interpretability.

  • Problem

    Healthcare diagnosis and treatment involve sequential decisions with sampled, evaluative, and delayed feedback, but evidence and methods for applying RL across these domains require systematic synthesis.

  • Method

    The survey organizes RL theoretical foundations and key techniques, reviews applications in treatment, diagnosis, critical care, and healthcare operations, and summarizes challenges and future directions.

  • Results

    RL applications are reported across dynamic treatment regimes, critical care, automated diagnosis, and other healthcare decision problems, with performance exceeding alternative techniques in many cases.

  • Takeaways & Limitations

    RL provides a framework for healthcare decision problems involving noisy, multidimensional, incomplete data, complex dynamics, and delayed evaluation feedback.

Abstract

from arXiv · show

As a subfield of machine learning, reinforcement learning (RL) aims at empowering one's capabilities in behavioural decision making by using interaction experience with the world and an evaluative feedback. Unlike traditional supervised learning methods that usually rely on one-shot, exhaustive and supervised reward signals, RL tackles with sequential decision making problems with sampled, evaluative and delayed feedback simultaneously. Such distinctive features make RL technique a suitable candidate for developing powerful solutions in a variety of healthcare domains, where diagnosing decisions or treatment regimes are usually characterized by a prolonged and sequential procedure. This survey discusses the broad applications of RL techniques in healthcare domains, in order to provide the research community with systematic understanding of theoretical foundations, enabling methods and techniques, existing challenges, and new insights of this emerging paradigm. By first briefly examining theoretical foundations and key techniques in RL research from efficient and representational directions, we then provide an overview of RL applications in healthcare domains ranging from dynamic treatment regimes in chronic diseases and critical care, automated medical diagnosis from both unstructured and structured clinical data, as well as many other control or scheduling domains that have infiltrated many aspects of a healthcare system. Finally, we summarize the challenges and open issues in current research, and point out some potential solutions and directions for future research.

I. INTRODUCTION

RL learns policies through trial-and-error interaction and evaluative feedback, making it suited to prolonged, sequential healthcare decisions. The survey organizes RL foundations and reviews applications across healthcare domains.

  • RL foundations: RL agents choose actions from current states, receive feedback and new states, and learn policies that maximize accumulated reward over time.This adaptive closed-loop process differs from supervised learning based on direct correct labels.
  • Healthcare motivation: Sequential treatment decisions can adapt treatment type, dosage, or re-examination timing to current health status and prior treatment history.The objective is to promote long-term patient benefits.
  • Survey scope: Existing healthcare studies have reported performance exceeding alternative techniques in many cases.The survey presents these studies as evidence of successful RL applications rather than a universal guarantee.
  • Survey scope: The survey covers RL applications in chronic-disease and critical-care treatment regimes, automated diagnosis, and healthcare resource allocation or scheduling.It also discusses challenges, open issues, and future research directions.
  • RL foundations: The paper structures RL foundations around general decision-making frameworks, model-based and model-free solutions, and efficiency- or representation-oriented techniques.The introductory framework includes Markov decision processes and value functions.

2) Basic Solutions and Challenging Issues:

RL solutions range from model-based dynamic programming to model-free Monte Carlo and temporal-difference methods, with policy-search and actor-critic alternatives. Core challenges include exploration, complexity, safety, robustness, and interpretability.

  • Basic solutions: Model-based methods require known transition and reward functions, whereas model-free methods learn policies from observations and rewards.Dynamic programming is model-based; Monte Carlo and temporal-difference methods are direct model-free approaches.
  • Basic solutions: Dynamic programming alternates value updates or policy evaluation and improvement to derive an optimal policy.Policy iteration begins with a policy, computes its value function, and then improves the policy.
  • Challenging issues: Dynamic programming requires a complete and accurate environment model, which is unrealistic in most applications.When the model is unavailable, direct interaction-based reinforcement learning is used instead.
  • Basic solutions: Temporal-difference methods update step by step, generally using less memory and computation than Monte Carlo methods, which update after episodes.Both methods estimate values from experienced returns, but differ in update timing.
  • Basic solutions: Q-learning is an off-policy temporal-difference algorithm, while actor-critic methods maintain explicit policy and value-function representations.Actor-critic methods are described as useful for large or continuous action spaces and stochastic policies.
  • Challenging issues: Exploration must balance exploiting known actions with exploring unknown actions, while healthcare RL also faces complexity, safety, robustness, and interpretability issues.The survey identifies these as fundamental or related challenges in RL research.

B. Key Techniques in RL

The survey groups contemporary RL techniques by whether they improve learning efficiency or enrich policy and value representations. These categories can overlap.

  • Key techniques: Contemporary RL techniques emphasize more sophisticated use of samples, environment models, prior-task knowledge, and learned representations.The survey notes that efficiency and representation classifications are not mutually exclusive.

1) Efficient Techniques:

Efficient RL techniques improve learning through experience reuse, environment modeling, and transfer across tasks. These approaches target learning efficiency, sample use, computation, and generalization.

  • Efficient techniques: Efficient techniques aim to improve convergence, sample efficiency, computational cost, or generalization by exploiting experience, models, or prior tasks.The survey distinguishes experience-, model-, and task-level techniques.
  • Experience-level: Experience-level methods store and reuse transition experience, supporting more stable and data-efficient batch reinforcement learning.Batch RL separates experience reuse from function approximation through supervised fitting.
  • Model-level: Model-level methods learn approximate transition and reward models, then plan multi-step actions using those models.They are described as normally more sample-efficient than model-free methods when the learned model is sufficiently accurate.
  • Task-level: Task-level transfer reinforcement learning transfers knowledge from source tasks to target tasks to reduce samples for near-optimal performance or increase final convergence.Transfer is framed as improving learning performance on relevant new tasks.

2) Representational Techniques:

Representational techniques make RL applicable to complex healthcare problems by compactly modeling policies, value functions, rewards, tasks, and partially observable states. Deep learning, multi-objective and preference-based rewards, and hierarchical representations address different sources of complexity.

  • Representation for Value Functions or Policies: Representational techniques compactly encode policies and value functions for continuous or highly dimensional state and action spaces.Linear approximations offer simpler implementation and stronger convergence guarantees, whereas nonlinear methods can avoid requiring highly informative features.
  • Representation for Value Functions or Policies: Deep neural networks support Deep RL by automatically extracting high-level features and semantic interpretations from input data.This reduces reliance on complex feature engineering and task-specific hand-crafted feature selection.
  • Representation for Reward Functions: Multi-objective RL trades off distinct reward signals to seek Pareto-optimal policies, while preference-based RL learns from qualitative ranking feedback.These approaches address settings where multiple objectives or nonnumeric feedback make a single scalar reward inadequate.
  • Representation for Tasks or Models: Factored MDPs compactly represent large structured transition models, and hierarchical RL decomposes complex tasks into smaller subtasks.Hierarchical decomposition can potentially reduce computational complexity.
  • Representation for Tasks or Models: Partially Observable RL is suited to healthcare systems because environmental states are often only partially observable or observations are noisy.Learning is more difficult because observations must be mapped to hidden states under additional uncertainty.

IV. DYNAMIC TREATMENT REGIMES

Dynamic treatment regimes frame individualized healthcare as sequential decision making, making RL suitable for adapting treatments to patient states and delayed outcomes. Applications span chronic disease and critical care, with cancer studies addressing flexible treatment stages, censoring, simulation, and radiotherapy scheduling.

  • Dynamic Treatment Regimes: Dynamic treatment regimes map clinical observations and treatment options to RL states and actions, while treatment outcomes correspond to reward functions.Their sequential decision rules are equivalent to RL policies.
  • Dynamic Treatment Regimes: RL can tailor time-dependent treatment decisions to patient heterogeneity without requiring accurate mathematical models or explicit treatment-outcome relationships.Reward engineering can also encode trade-offs between treatment efficacy and side effects.
  • Cancer: Cancer studies apply Q-learning, actor-critic methods, preference-based RL, and censored-Q-learning using clinical, simulated, or computational data.The survey organizes examples by application scenario, RL method, learning data, and study highlights and limits.
  • Cancer: Preliminary radiotherapy findings suggest non-uniform fractionation schedules may benefit some tissue types.The comparison involved exhaustive policy search, FQI, SARSA(λ), and K-Nearest Neighbors-TD(λ).
  • Cancer: Multi-objective and deep RL methods have been used to optimize radiotherapy dose and automate adaptive decisions for NSCLC.Distributed Q-learning targets Pareto-optimal dose solutions, while a multi-component framework combines GAN, transition DNN, and DQN components.
  • Cancer: Cancer DTRs face censoring from treatment dropout and variable numbers and timing of therapy lines across patients.These issues complicate individualized optimization; NSCLC is cited as an example with one to three treatment lines.

2) Diabetes:

Diabetes treatment requires personalized glucose regulation because patient variability, disturbances, and modeling difficulty complicate insulin control. RL studies therefore use model-free, model-based, and data-driven approaches for adaptive insulin dosing and treatment regimes.

  • Diabetes: Patient variability, disturbances such as meals and exercise, and difficulty modeling glucose-insulin dynamics motivate personalized adaptive glucose regulation.The context concerns artificial-pancreas systems using continuous glucose monitoring and closed-loop control.
  • Diabetes: RL has been applied to artificial-pancreas control, including model-free Q-learning for insulin delivery without an explicit glucose-insulin model.An initial study targeted normoglycemia around 80 mg/dl.
  • Diabetes: Model-based VI, actor-critic, and SARSA methods have been evaluated with mathematical glucose-insulin models for fasting and post-meal regulation.The cited studies use the Palumbo, Bergman, and Hovorka models.
  • Diabetes: Data-driven studies estimate treatment values or policies from clinical records and mobile observations, including doctor-defined and inverse-learned rewards.V-learning was used to estimate regimes intended to reduce hypo- and hyperglycemic episodes with minimal assumptions about data generation.

3) Anemia:

RL applications in anemia, HIV, and mental disease address individualized sequential treatment under patient variability, uncertainty, and difficult-to-model dynamics. Reported examples include improved anemia dosing, clinically compliant HIV recommendations, and optimized epilepsy stimulation and depression treatment learning.

  • Anemia: Anemia management requires individualized ESA dosing because patient responses vary and standard protocols can leave hemoglobin oscillating around the target range.Clinicians otherwise assess monthly hemoglobin and iron levels before adjusting doses.
  • Anemia: 27.6% more patients reached the targeted hemoglobin range while drug quantity fell 5.13% with FQI in a computational-model evaluation.The study evaluated ESA dosing strategies for anemia treatment.
  • HIV: HIV RL studies learned structured treatment interruption and other drug-prescription strategies using simulated and real clinical data.A ten-year evaluation involving 250 HIV patients found recommendations reasonably compliant with clinician-suggested treatments.
  • HIV: HIV treatment evaluation must account for patient immune-system variation and uncertainty across subpopulations.The survey also describes exploration difficulties because the healthy steady-state has a relatively small basin of attraction.
  • Mental Disease: RL optimized epilepsy stimulation by adapting policies to neural dynamics, reducing seizure incidence by 25% and electrical stimulation by about a factor of 10.The study used EEG features and stimulation-frequency actions from an in vitro rat-brain model.
  • Mental Disease: Kernel-based BRL and Q-learning extensions were applied to depression treatment, with IQ-learning improving on Q-learning in integrated mean squared error.The studies used STAR*D data and addressed nonsmooth decision rules and nonregular parameter estimation.

B. Critical Care

Critical care applications use RL to derive treatment policies from rich but noisy ICU data, especially for sepsis management. Reported studies address medication decisions and suggest improved outcomes in some settings.

  • B. Critical Care: ICU data include free-text notes, images, physiological waveforms, and vital-sign time series, but are noisy, biased, and incomplete.The survey identifies data processing and interpretation as a premier critical-care challenge.
  • B. Critical Care: Critical-care RL applications include sepsis, sedation regulation, mechanical ventilation, and heparin dosing.The survey organizes these applications by RL technique and data source in Table IV.
  • 1) Sepsis:: RL studies have addressed sepsis treatment decisions using databases such as MIMIC, despite the lack of universally agreed-upon decision support.Applications include deducing treatment policies for patients with sepsis.
  • 1) Sepsis:: Continuous-state modeling identified interpretable sepsis policies and potentially reduced hospital mortality by 1.8 - 3.6%.The approach used a fully-connected Dueling Double DQN with Double DQN, Dueling DQN, and Prioritized Experience Replay.
  • 1) Sepsis:: DDPG with potential-based reward shaping achieved 0.8% mortality across 500 simulated patient parameterizations, versus average mortality of 49%.The study targeted personalized multi-cytokine mediation therapy in an agent-based sepsis model.

2) Anesthesia:

RL has been applied to ICU anesthesia and sedation control, where dosing must maintain physiological stability over extended periods. Studies report improved control, personalization, and dose efficiency relative to conventional or fixed controllers.

  • 2) Anesthesia:: ICU sedation is challenging because mechanically ventilated patients may require adequate sedation for several days.The goal is to maintain physiological stability while reducing patient pain.
  • 2) Anesthesia:: RL anesthesia research includes model-based control, Q-learning, fuzzy controllers incorporating physician experience, and inverse reinforcement learning from anesthesiologist trajectories.These approaches span simulation, personalization, and clinical study settings.
  • 2) Anesthesia:: RL anesthesia control using BIS as a controlled variable improved individualized management and patient outcomes compared with traditional controlled administration.BIS is used as a surrogate measure of anesthetic effect.
  • 2) Anesthesia:: TD Q(λ) demonstrated superior stability and responsiveness to a well-tuned PID controller for intravenous propofol administration.The state included BIS error and estimated four-compartment propofol concentrations, while propofol dose served as the control action.
  • 2) Anesthesia:: A continuous actor-critic learner reduced administered anesthetic dose by 9.4% versus a fixed controller while keeping BIS error clinically acceptable 93.9% of the time.The method first learned a generic strategy from average patient data and then fine-tuned it to individual patients.

V. AUTOMATED MEDICAL DIAGNOSIS

Automated medical diagnosis uses RL to handle sequential decisions over structured and unstructured clinical data. The surveyed applications span medical image analysis, diagnostic testing, symptom checking, and conversational diagnosis.

  • V. AUTOMATED MEDICAL DIAGNOSIS: Diagnostic error accounts for as high as 10% of deaths and 17% of adverse events in hospitals, motivating automated diagnostic support.Diagnosis requires clinicians to assimilate complex and diverse clinical reports.
  • V. AUTOMATED MEDICAL DIAGNOSIS: Existing supervised diagnostic methods rely heavily on annotated samples and have limits capturing diagnosing-process dynamics and uncertainties.Clinical inputs are commonly divided into structured medical data and unstructured narrative text.
  • A. Structured Medical Data: RL in structured-data diagnosis is especially prominent for medical-image feature extraction, segmentation, and object detection, with DQN achieving more efficient, accurate, and robust landmark detection.Reported evaluations covered 2D MRI, ultrasound, and 3D CT images, with later extensions to multi-scale and incomplete data.
  • A. Structured Medical Data: An RL approach optimized diagnostic testing for solitary pulmonary nodules using a multi-objective criterion covering cost, morbidity, mortality, and time expense.Its evaluation reported improved testing strategies compared with several fixed testing strategies.
  • B. Unstructured Medical Data: Symptom-checking RL formulated inquiry and diagnosis as an MDP and substantially improved disease-prediction accuracy on simulated data.The resulting system was employed in the DeepQ Tricorder, which won second prize in the 2017 Qualcomm Tricorder XPRIZE competition.
  • B. Unstructured Medical Data: RL dialogue systems collected symptoms conversationally and significantly outperformed supervised learning for mild cognitive impairment diagnosis using only a few turns.Another system trained dialogue policies to maximize diagnosis accuracy while minimizing conversation turns.

VI. OTHER HEALTHCARE DOMAINS

RL has been applied beyond treatment regimes and diagnosis to healthcare scheduling, process control, drug discovery, and health management. Reported applications include resource allocation, adaptive control, molecule generation, and personalized interventions.

  • Health Resource Scheduling and Allocation: RL formulates healthcare resource allocation as sequential decision making and has been applied to radiology CT-scan scheduling.Approaches include Q-learning for MDP-based allocation and simulation-based approximate dynamic programming under stochastic service times and uncertain arrivals.
  • Optimal Process Control: RL has been used to learn control policies for surgical robots, functional electrical stimulation, and adaptive medical video streaming.Applications include robot operation, stimulation-pattern adaptation, and Q-learning-based rate control for bandwidth-demanding ultrasound video.
  • Drug Discovery and Development: Drug discovery uses computational molecule design to search large spaces for compounds satisfying criteria such as bioactivity and synthetic accessibility.The discovery process typically lasts 10-17 years and has around 10% overall probability of success; deep RL has generated targeted chemical libraries with desired properties.
  • Health Management: RL-derived personalized interventions increased activity and walking pace in 27 sedentary type 2 diabetes patients, while static policies did not.The RL group also experienced a superior reduction in blood glucose, with longer participation associated with greater reductions.
  • Scope and Limitations: Despite successes, many healthcare studies apply naive RL methods in relatively simplified settings, leaving common practical limitations.The survey frames these limitations as challenges for advancing applications across the reviewed domains.

A. State/Action Engineering

Healthcare RL depends on state, action, and reward formulations that preserve clinically relevant information and objectives. Current approaches face biased or incomplete data, discretized actions, difficult reward specification, and limited use of preference-based or inferred rewards.

  • State Engineering: Medical data contain clinician-, device-, and institution-varying biases and noise, while missing or censored observations complicate state representation and increase value-estimate variance.Censoring is more challenging than missingness because it requires more sophisticated state representation and value estimation.
  • State Engineering: Most existing state representations use raw physiological, pathological, and demographic information, but may omit the temporal dependence and causal information central to medical data.These representations commonly rely on discretization or function approximation with linear or deep neural models.
  • Action Engineering: Discretizing actions into limited bins is inadequate for settings requiring continuous or multidimensional decisions such as precision dosing.Continuous action selection remains non-trivial in large or infinite action spaces, despite progress in actor-critic and policy-search methods.
  • Reward Formulation: Healthcare RL commonly uses explicitly defined numerical rewards, but specifying them precisely can be difficult or misleading when treatment objectives conflict.Cancer-treatment examples decompose objectives into weighted components and require threshold and weighting parameters to trade off efficacy and safety.
  • Reward Formulation: Preference-based and multi-objective RL have limited medical applications and generally consider static preferences or fixed objectives.Medical rewards may instead change with time, clinical situation, and patients’ evolving physiopsychic conditions.
  • Reward Formulation: Healthcare researchers may infer rewards from expert treatment trajectories rather than specify them directly.Inverse reinforcement learning is identified as an approach for estimating expert rewards from retrospective medical data.
  • Reward Formulation: Long-term outcomes observed only at episode end create sparse-reward challenges, while ignoring short-term rewards can prevent learning important state-action relationships.Existing healthcare studies addressing sparse rewards mainly focus on dynamic treatment regimes with typically three or four steps.

C. Policy Evaluation

Healthcare policy evaluation is constrained because target policies cannot usually be tested directly on patients. Retrospective off-policy evaluation is therefore necessary but vulnerable to state-representation errors, estimator variance, confounding, and weak exploration strategies.

  • Policy Evaluation: Off-policy evaluation estimates a target policy’s value using data collected by a different behavior policy.In healthcare, retrospective evaluation is needed before testing learned policies in clinical environments.
  • Policy Evaluation: Direct policy testing on patients is often infeasible because experiments are costly, risky, or ethically and legally constrained.These constraints make retrospective data important for preliminary assessment of learned treatment policies.
  • Policy Evaluation: Retrospective policy estimates can be unreliable or misleading when state representation, importance-sampling variance, and confounders are handled inappropriately.The survey highlights sepsis management as an example of the fundamental difficulty of evaluation using retrospective health data.
  • Exploration Strategies: Most healthcare applications use simple heuristic exploration such as ε-greedy, which can be inefficient in complex or large state-action spaces.Naive exploration is especially problematic when only a small portion of the state space is reachable or when actions and states are continuous.
  • Exploration Strategies: Exploration can be unacceptable when inappropriate actions cause consequences beyond poor performance.The healthcare setting therefore requires attention to the true cost of exploratory actions, not only their reinforcement penalty.
  • Credit Assignment: Delayed and variable treatment effects make temporal credit assignment particularly prominent in healthcare.Standard heuristics such as eligibility traces and discount factors use elapsed time to weight past actions or future events.
  • Future Directions: Addressing these challenges is important for adopting RL-based medical procedures and clinical strategies.The survey advocates combining advances in RL theory and techniques with emerging clinical requirements, including ambient intelligence and real-life applications.

A. Interpretable Strategy Learning

Interpretability, limited data, and the need to incorporate clinical knowledge remain important barriers to healthcare RL. Proposed directions include interpretable policy representations, transfer and interactive learning, data-scarcity methods, and ambient-intelligence applications.

  • Interpretable Strategy Learning: RL policies expressed through black-box models can be difficult to interpret, limiting transparency in healthcare decision making.The survey identifies interpretability as a profound issue for modern machine learning methods, including RL.
  • Interpretable Strategy Learning: Interpretable RL can use index-based formulas, symbolic program synthesis, genetic programming, or compact algebraic policy representations.These approaches target more transparent representations of learned policies.
  • Knowledge Integration: Healthcare prior knowledge can enter RL through model configuration, transfer across patients or domains, and human-in-the-loop learning.The survey presents these as mechanisms for improving learning performance and incorporating domain expertise.
  • Knowledge Integration: Monotonic dose-response knowledge can prevent exploration of lower doses after an insufficient response to a particular dose.A modified Q-learning update incorporated this clinical constraint for anemia management involving HGB and EPO dosing.
  • Knowledge Integration: Transfer learning remains limited in healthcare, although pretrained parameters, latent-variable models, and task mappings may support related personalized-care tasks.The intended scope includes groups of patients with similar diagnoses.
  • Data Scarcity: Many healthcare domains lack sufficient training samples, especially for new diseases and rare illnesses with few accurately labeled cases.Applying RL directly to limited data may produce overly optimistic or pessimistic treatment assessments.
  • Data Scarcity: Data scarcity can be addressed through augmentation or model and domain-adaptation methods such as GANs, knowledge distillation, and meta-learning.These approaches either increase available samples or modify learning to improve efficiency with limited data.
  • Ambient Intelligence: Ambient-intelligence healthcare combines pervasive sensing with online RL to adapt interventions continuously to user needs.Reported examples include mobile interventions recommending physical activity for people with diabetes or cardiac rehabilitation experience.

E. Future in-vivo Studies

Future healthcare applications of RL require in-vivo validation because existing work largely relies on computational patient models or retrospective clinical data. Real-world deployment must address safety, sample efficiency, robustness, data challenges, and inter-individual physiological differences.

  • Future in-vivo Studies: Most healthcare RL studies rely on mathematical patient-response models or retrospective clinical data rather than in-vivo evaluation.These approaches support early exploration and direct derivation of adaptive treatment strategies, but do not reliably assess real-life performance and personalization.
  • Future in-vivo Studies: In-vivo studies are urgently needed to evaluate RL performance and personalization in real-life healthcare implementations.The survey identifies closed-loop evaluation with real patients as an important next step beyond in-silico research.
  • Future in-vivo Studies: Safety is paramount because incorrect healthcare actions can produce long-term or unrecoverable effects that later exploitation cannot compensate for.This makes safe action selection during learning essential when RL is applied to in-vivo subjects.
  • Future in-vivo Studies: Sample efficiency remains a major constraint because collecting many real-patient samples is costly and repeated trial-and-error treatment may be unsustainable.Batch learning can use past samples more efficiently, while model-based methods are identified as another promising direction.
  • Future in-vivo Studies: Robust deployment must handle incomplete or noisy states in partially observable environments and provide confidence measures for derived solutions.The survey specifically mentions standard errors, confidence sets, and hypothesis tests as examples of needed uncertainty measures.
  • Future in-vivo Studies: Progress requires integrating RL expertise with clinical involvement while carefully addressing safety, robustness, and efficiency in real-life applications.The paper frames healthcare RL as cross-disciplinary research involving machine learning researchers and clinicians directly involved in patient care.
Loading 1908.08796v4…