Source-linked AI summary
Supervised Reinforcement Learning with Recurrent Neural Network for Dynamic Treatment Recommendation
Lu Wang, Wei Zhang, Xiaofeng He, Hongyuan Zha
TL;DR
Treatment recommendation must balance matching clinicians’ prescriptions with optimizing patient outcomes, because supervised and reinforcement learning address different parts of this problem. SRL-RNN combines both signals in an off-policy actor-critic model with recurrent processing, and experiments on MIMIC-3 report up to a 4.4% reduction in estimated in-hospital mortality alongside better medication recommendation.
Problem
Existing treatment-recommendation studies separately match doctors’ prescriptions or optimize evaluation signals, while the ground truth of good treatment remains unclear and unsupervised RL may create unacceptable risks.
Method
SRL-RNN jointly uses indicator and evaluation signals in an off-policy actor-critic architecture with RNN processing for dynamic treatment recommendation.
Results
4.4%: Experiments on MIMIC-3 report up to a 4.4% reduction in estimated in-hospital mortality and better medication recommendation.
Takeaways & Limitations
Combining clinician supervision with outcome-based reinforcement learning supports a prescription policy that addresses both safety and treatment outcomes within the evaluated MIMIC-3 setting.
Takeaways & Limitations
Estimated mortality does not equal mortality in real clinical settings, although it is used as a computational-testing metric.
Abstract
from arXiv · showhide
Dynamic treatment recommendation systems based on large-scale electronic health records (EHRs) become a key to successfully improve practical clinical outcomes. Prior relevant studies recommend treatments either use supervised learning (e.g. matching the indicator signal which denotes doctor prescriptions), or reinforcement learning (e.g. maximizing evaluation signal which indicates cumulative reward from survival rates). However, none of these studies have considered to combine the benefits of supervised learning and reinforcement learning. In this paper, we propose Supervised Reinforcement Learning with Recurrent Neural Network (SRL-RNN), which fuses them into a synergistic learning framework. Specifically, SRL-RNN applies an off-policy actor-critic framework to handle complex relations among multiple medications, diseases and individual characteristics. The "actor" in the framework is adjusted by both the indicator signal and evaluation signal to ensure effective prescription and low mortality. RNN is further utilized to solve the Partially-Observed Markov Decision Process (POMDP) problem due to the lack of fully observed states in real world applications. Experiments on the publicly real-world dataset, i.e., MIMIC-3, illustrate that our model can reduce the estimated mortality, while providing promising accuracy in matching doctors' prescriptions.
1 INTRODUCTION
Treatment recommendation research has developed supervised and reinforcement-learning approaches, but each leaves an important clinical need unresolved. SRL-RNN combines doctor-prescription supervision, outcome-based evaluation, and recurrent modeling for dynamic treatment recommendation.
- Supervised prescription models match doctor prescriptions, but the ground truth of a good treatment strategy remains unclear.
- Reinforcement learning optimizes long-term patient outcomes in dynamic treatment regimes, but may recommend treatments substantially different from doctors’ prescriptions.
- Figure 1 illustrates that medications vary over time as patient observations accrue during hospitalization.
- SRL-RNN integrates indicator and evaluation signals within an off-policy actor-critic architecture for multiple diseases and medications.
- The model further uses an RNN to address the partially observed state problem in real-world treatment recommendation.
- 4.4%: SRL-RNN reduces estimated in-hospital mortality while also providing better medication recommendation on MIMIC-3.
2 RELATED WORK
Related work centers on supervised learning, reinforcement learning, and methods that combine expert behavior with reward optimization. Existing approaches face uncertainty about treatment quality, safety concerns, and difficulty handling complex medication spaces.
- Supervised learning minimizes differences from doctors’ prescriptions, while reinforcement learning maximizes cumulative rewards such as disease assessments or survival rates.
- Supervised-learning methods remain limited because doctors’ behaviors may be imperfect and the ground truth of good treatment is unclear.
- Clinical reinforcement-learning methods have been studied for schizophrenia, non-small cell lung cancer, and ICU heparin dosing, including under POMDP conditions.
- Without knowledgeable supervision, reinforcement-learning systems may recommend treatments significantly different from doctors’ prescriptions, creating unacceptable risks.
- Value-based reinforcement-learning methods are difficult to apply to multiple diseases and complex medication spaces.
- Imitation learning and supervised actor-critic methods incorporate expert behavior, but prior approaches train their signal components separately or rely on expert-optimality assumptions.
3 BACKGROUND
The paper formulates dynamic treatment recommendation as a finite-horizon decision problem and reviews value-based, policy-gradient, and actor-critic reinforcement learning. These preliminaries motivate off-policy actor-critic modeling for complex treatment decisions.
- 3.1 Problem Formulation: Dynamic treatment recommendation is modeled as a finite-horizon MDP with action space A, state space S, and reward function r: S×A → R.
- 3.1 Problem Formulation: The learned policy selects medications from patient observations while maximizing discounted cumulative rewards and minimizing differences from clinician decisions.
- 3.2 Model Preliminaries: Value-based reinforcement learning learns greedy policies from action-value functions, whereas policy-gradient methods optimize parameterized stochastic or deterministic policies.
- 3.2 Model Preliminaries: DQN estimates nonlinear Q functions using replay buffers and asynchronously updated target networks.
- 3.2 Model Preliminaries: Actor-critic methods combine an actor that optimizes the policy with a critic that estimates action values.
- 3.2 Model Preliminaries: Off-policy actor-critic learning estimates the target policy’s value using state distributions generated by a behavior policy, while DDPG uses soft target-network updates.
4 SRL-RNN ARCHITECTURE
SRL-RNN combines supervised and reinforcement learning in an off-policy actor-critic architecture, using recurrent processing to recommend dynamic treatments from partially observed patient states. The actor balances doctors’ prescriptions with predicted treatment value, while the critic guides training and is not required at test time.
- Overview: SRL-RNN combines supervised learning and reinforcement learning to recommend tailored treatments while balancing prescription matching and expected clinical return.The objective uses a weight parameter ϵ to trade off the reinforcement-learning and supervised-learning tasks.
- Actor Network: The actor maps patient states to time-varying medication prescriptions and is jointly optimized using reinforcement-learning and supervised-learning objectives.The supervised component minimizes differences from doctors’ prescriptions, while the reinforcement-learning component maximizes expected return.
- Actor Network Update: The actor’s reinforcement-learning gradient uses the critic’s action-value gradient to move medication predictions toward actions judged better and away from actions judged worse.The actor is updated through the chain rule using ∇aQw(s,a) and the policy Jacobian.
- Actor Network Update: The supervised objective uses cross-entropy over medication categories, where doctor choices provide binary targets and the actor outputs medication probabilities.K denotes the number of medications or medication categories.
- Critic Network: The critic estimates the action-value function from patient states, prescriptions, actor outputs, and rewards, then guides actor updates toward performance improvement.The critic is used during training, whereas only the actor is required at test time.
- Recurrent Processing: LSTM summarizes historical observations so SRL-RNN can operate when patient states are not fully observable in the POMDP formulation.The learning procedure stores observation-action-reward sequences and initializes actor, target-actor, critic, and target-critic networks.
5 EXPERIMENTS
Experiments on MIMIC-3 evaluate mortality, prescription consistency, feature contributions, stability, and qualitative recommendations. SRL-RNN generally outperforms baselines by combining supervised and reinforcement-learning signals in dynamic treatment recommendation.
- Experimental Setup: MIMIC-3 contains 43K critical-care patients, 6,695 diseases, and 4,127 drugs; experiments retain the top 1,000 medications and 2,000 diseases.Patient representations include demographic, disease, laboratory, vital-sign, and output-event features.
- Evaluation Metrics: Estimated in-hospital mortality is computed from discretized learned Q-values, while mean Jaccard measures consistency with doctors’ medication sets.The paper notes that estimated mortality is not equal to real clinical mortality but is used as a computational-testing metric.
- Model Comparisons: RL outperforms supervised baselines, SD3Q outperforms D3Q, and LEAP outperforms POP-20 in their respective comparisons.These comparisons support benefits from evaluation feedback, knowledgeable supervision, and modeling relations among diseases and medications.
- Model Comparisons: SRL-RNN performs significantly better than all adopted baselines in both dynamic and static treatment settings.The authors attribute this to sequential decision modeling, evaluation feedback, doctor-prescription supervision, and an off-policy actor-critic framework.
- Ablation Study: Adding patient-specific features monotonically increases Jaccard scores, while SRL-RNN’s estimated mortality monotonically decreases.The reported feature groups are time-series variables, diseases, and demographics.
- Policy Analysis: At the minimum prescription difference, mortality rates are 0.021 for RL and 0.016 for SRL-RNN.The comparison indicates lower observed mortality for SRL-RNN at this point.
- Policy Stability: SRL-RNN combines reinforcement and supervised learning stably; Jaccard is learned faster than Q-values, whose learning may require rich trajectories.Figure 5 examines expected return and Jaccard across learning epochs and feature settings.
- Hyperparameter Analysis: The highest Jaccard scores and lowest mortality rates occur at ϵ values of 0.5 and 0.6.ϵ balances reinforcement learning and supervised learning in the reported experiment.
6 CONCLUSION
The paper introduces SRL-RNN for dynamic treatment recommendation by jointly using supervised and reinforcement learning, with recurrent modeling for partially observed clinical states. Experiments on real-world EHR data show reduced estimated hospital mortality and improved medication recommendation.
- SRL-RNN combines indicator and evaluation signals through joint supervised and reinforcement learning for dynamic treatment recommendation.
- The model uses an off-policy actor-critic architecture to discover optimal dynamic treatments and an RNN to address the POMDP problem.
- SRL-RNN reduces estimated hospital mortality by up to 4.4% while providing better medication recommendation on a real-world EHR dataset.