Source-linked AI summary
Evaluating Reinforcement Learning Algorithms in Observational Health Settings
Omer Gottesman, Fredrik Johansson, Joshua Meier, Jack Dent, Donghun Lee, Srivatsan Srinivasan, Linying Zhang, Yi Ding, David Wihl, Xuefeng Peng, Jiayu Yao, Isaac Lage, Christopher Mosch, Li-wei H. Lehman, Matthieu Komorowski, Matthieu Komorowski, Aldo Faisal, Leo Anthony Celi, David Sontag, Finale Doshi-Velez
TL;DR
Healthcare RL policies are increasingly learned from observational data, but evaluating them before high-stakes clinical use is difficult. The paper analyzes how state representations, estimator variance, and confounding can produce unreliable policy estimates, then proposes thoughtful evaluation and domain-guided safeguards.
Problem
Before black-box RL treatment policies are used in high-stakes clinical decisions, their quality must be evaluated from observational data without directly experimenting on patients.
Method
The paper provides a conceptual analysis of RL evaluation, using sepsis management to examine state representation, off-policy evaluation, estimator variance, and confounding.
Results
The paper shows that data and methodological artifacts can yield clinically implausible policies and unreliable value estimates, including high variance when deterministic evaluation policies differ from behavior policies.
Takeaways & Limitations
Evaluation should examine representation adequacy, importance-weight distributions, and causal information with guidance from domain experts before clinical adoption.
Takeaways & Limitations
Off-policy estimators may have very few informative samples and high variance when the evaluation and behavior policies differ substantially.
Abstract
from arXiv · showhide
Much attention has been devoted recently to the development of machine learning algorithms with the goal of improving treatment policies in healthcare. Reinforcement learning (RL) is a sub-field within machine learning that is concerned with learning how to make sequences of decisions so as to optimize long-term effects. Already, RL algorithms have been proposed to identify decision-making strategies for mechanical ventilation, sepsis management and treatment of schizophrenia. However, before implementing treatment policies learned by black-box algorithms in high-stakes clinical decision problems, special care must be taken in the evaluation of these policies. In this document, our goal is to expose some of the subtleties associated with evaluating RL algorithms in healthcare. We aim to provide a conceptual starting point for clinical and computational researchers to ask the right questions when designing and evaluating algorithms for new ways of treating patients. In the following, we describe how choices about how to summarize a history, variance of statistical estimators, and confounders in more ad-hoc measures can result in unreliable, even misleading estimates of the quality of a treatment policy. We also provide suggestions for mitigating these effects---for while there is much promise for mining observational health data to uncover better treatment policies, evaluation must be performed thoughtfully.
1 Motivation
The paper examines how to evaluate reinforcement-learning treatment policies using observational healthcare data before deploying them in high-stakes settings. It highlights representation choices, estimator variance, and confounding as sources of unreliable policy-quality estimates.
- Motivation: Observational evaluation is needed because researchers seek evidence that proposed treatment strategies may outperform current practice without experimenting directly on patients.The paper frames this as a common healthcare setting for evaluating RL policies learned from historical data.
- Motivation: RL evaluation can be misleading when history summaries, statistical estimators, or ad-hoc measures fail to preserve relevant information or control uncertainty.The paper specifically identifies history summarization, estimator variance, and confounders as sources of unreliable or misleading estimates.
- Motivation: The paper offers a conceptual starting point for clinical and computational researchers designing and evaluating algorithms for new treatment strategies.
2 Running Example: Managing Sepsis in the ICU
The paper uses sepsis management in the ICU to formalize an RL problem around sequential treatment decisions. It defines patient states, discretized treatment actions, survival-based rewards, and off-policy evaluation from historical data.
- Running Example: Managing Sepsis in the ICU: Sepsis management involves sequential decisions about IV fluids and vasopressors whose mortality effects may appear only days after treatment.Short-term targets such as blood pressure may not correlate strongly with mortality, making treatment evaluation difficult.
- Running Example: Managing Sepsis in the ICU: The example uses MIMIC III sepsis data with 47 features measured at 4-hour intervals, including treatments and 90-day mortality.The cohort contains 19,275 ICU patients meeting sepsis-3 criteria.
- Running Example: Managing Sepsis in the ICU: The RL formulation requires a state space, action space, and reward function, with the state summarizing a patient’s history.The paper notes that histories are unwieldy, so practitioners often replace them with simpler covariate summaries.
- Running Example: Managing Sepsis in the ICU: IV fluids and vasopressors are discretized into five bins each, producing 25 possible treatment choices per 4-hour interval.The first bin represents no treatment and the remaining bins represent physician-prescribed quartiles.
- Running Example: Managing Sepsis in the ICU: The reward is zero until the final interval, when it is positive for survival and negative otherwise.The policy objective is cumulative reward across the patient’s history.
- Running Example: Managing Sepsis in the ICU: Off-policy evaluation estimates a policy’s expected cumulative reward from historical data when directly trying that policy on patients is not feasible.The policy value is defined as expected cumulative reward under the policy.
3 Why do we need careful evaluation?
Illustrative sepsis examples show that RL agents can recommend clinically implausible actions when observational data or the available action space is misleading. These failures motivate careful evaluation for artifacts in data and methodology.
- Why do we need careful evaluation?: A standard RL algorithm recommended minimal treatment for patients with very high SOFA scores, despite the recommendation being clinically implausible.Because highly acute patients usually received aggressive treatment and had high mortality, the algorithm lacked evidence about untreated high-acuity patients.
- Why do we need careful evaluation?: With only intubation available, the RL agent recommended intubation more often than clinicians because it lacked alternative treatment choices.Adding mechanical ventilation, IV fluids, and vasopressors eliminated the observed overuse of mechanical ventilation.
- Why do we need careful evaluation?: These examples show that artifacts in data or methodology can lead AI algorithms to learn harmful policies that standard evaluation may fail to recognize.The paper therefore recommends examining when evaluation methods miss such artifacts and adopting best practices.
4 Challenges with Choosing Representations
Choosing a state representation requires compressing patient history while retaining information relevant to treatment decisions and outcomes. Missing confounders or insufficient summaries can distort policy evaluation, and their adequacy cannot be established from statistical quantities alone.
- Challenges with Choosing Representations: A state representation must include variables that affect both observed treatment decisions and outcomes, or confounding can reverse the apparent effect of treatment.For example, failing to adjust for acuity could make treatment appear to increase mortality even when the opposite is true.
- Challenges with Choosing Representations: Confounders may include a patient’s medical record, genetic makeup, and socioeconomic or environmental factors, and statistical quantities cannot verify that all have been measured.Domain knowledge is required to justify that important confounding factors have been identified.
- Challenges with Choosing Representations: Identifying confounders is difficult even for a single treatment decision and becomes more difficult in sequential decision settings.
- Challenges with Choosing Representations: All RL algorithms assume supplied features sufficiently summarize the patient history, but identifying an adequate representation is not straightforward.The paper raises whether recent dose, cumulative dose, or other history is needed to predict response to a vasopressor.
- Challenges with Choosing Representations: The paper illustrates representation sensitivity by clustering current measurements into 100 k-means groups and repeatedly learning policies from those clusters.This treats the patient’s current cluster as sufficient to summarize history, despite being a simplified representation.
- Challenges with Choosing Representations: When quantitative validation is unavailable, interpretability and consultation with expert clinicians are presented as fallback ways to assess representation validity.
5 Challenges with Statistical Evaluation
Off-policy evaluation in observational healthcare data can become unreliable because importance sampling relies on sparse trajectory matches, creating high variance and selection bias. In the sepsis case study, weighted estimators reduced variance but remained vulnerable to bias, motivating diagnostics and stochastic policies.
- Challenges: Only a tiny fraction of histories may receive non-zero importance weights when evaluating deterministic policies, producing high-variance estimates.Importance sampling retains trajectories whose observed actions match the evaluation policy; sparse outcomes further require matching entire sequences.
- Estimator behavior: WDR had lower variance than WPDIS at approximately the same mean, but its model component introduced optimistic bias.The model-based estimate had lower variance but was optimistically biased, while weighted importance sampling gave more realistic values than unweighted estimators.
- Policy comparisons: The WDR estimator suggested that the model-based optimal policy outperformed alternatives, but WPDIS showed larger variance and no clear dominance.The compared policies included the RL-based, physician, uniformly random, and never-treat policies; the figure reports noisy and inconclusive comparisons.
- Estimator failure modes: The WPDIS estimator made the no-treatment policy appear superior because deterministic-policy weights retained lower-risk untreated patients, creating selection bias.For deterministic policies, the importance-weight numerator becomes a product of zeros and ones, so only fully matching sequences remain.
- Diagnostics: In the sepsis data, the learned policy had an average effective sample size of 25 out of 3,855 patients, with selected stays averaging 3.7 intervals versus 13 overall.Even a deterministic policy closest to physicians used only 167 of 3,855 patients, whose average stay was six intervals.
- Mitigation: Evaluating non-deterministic policies can increase the fraction of observational data with non-zero weights and make importance-sampling estimates more robust.Allowing multiple actions per state to have non-zero recommendation probabilities also leaves room for clinician judgment.
6 Challenges With Ad-Hoc Evaluation Measures
Ad-hoc U-curve measures can make random and no-treatment policies appear effective because confounding and dosage binning produce misleading mortality patterns.
- Diagnosing and Mitigating Concerns: Because confounding and action binning can create these artifacts, researchers should generally avoid ad-hoc evaluation measures and seek expert face-validity checks instead.Alternative equal-width bins can leave severe-patient observations sparse.
- Illustration: The same U-shaped mortality curves appear for the learned, random, and no-treatment policies, so the curves do not establish that the learned policy is better.For vasopressors, the results for all three policies are indistinguishable.
- Why the curves mislead: The no-treatment curve primarily reflects the correlation between higher dosages and higher mortality among sicker patients.Higher-risk patients are more likely to receive higher treatment dosages.
- Why the curves mislead: Discretizing continuous dosages into physician-derived quartiles can make most bins nearly identical, rendering the learned policy’s dosage recommendation insignificant for many observations.The long-tailed dosage distribution makes the medians of all but the last quartiles very close.
7 Recommendations for Researchers
The recommendations emphasize causal data and representation, adequate support for evaluated actions, effective sample sizes, and expert interpretability throughout the RL workflow.
- Design data collection and representation to support causal conclusions: Causal conclusions require recording clinically relevant confounders and preserving them during model or variable selection.Dimensionality reduction can introduce confounding bias when confounders are omitted.
- Limit yourself to actions that were taken and policies that are similar to physicians’: Policies should stay within actions sufficiently represented in retrospective data and remain reasonably similar to physicians’ policies.Retrospective data cannot reliably evaluate treatment decisions that were rarely or never taken.
- Calibrate expectations and define goals: Available data and methods are better suited to recommendation systems or modest policy changes than to discovering unprecedented treatment strategies.The paper frames these uses as practical goals under current limitations.
- Be cognizant of effective sample sizes: Importance-sampling evaluations must be checked for adequate effective sample sizes because deterministic policies can retain only a small fraction of the cohort.The number of non-zero-weight sequences can decay exponentially with sequence length.
- Interpretability and expert evaluation: Expert evaluation should cover state representations, reward design, final policies, and the features driving policy differences.Clinicians should assess whether short-term optimized goals correspond to desirable long-term outcomes.
8 Conclusions
The paper argues that retrospective evaluation of sequential healthcare decisions remains fundamentally difficult. It calls for careful evaluation and continued development of best practices that preserve the promise of observational health data.
- Conclusions: Retrospective evaluation of sequential decisions is fundamentally hard, so researchers should engage deeply with both its promise and its challenges.The authors present the article as raising more questions than it answers.