Source-linked AI summary
Offline Reinforcement Learning for Hemodynamic Management of Sepsis in the ICU: a MIMIC-IV Study with Dual Off-Policy Evaluation
Marc Pérez-Roig, David Fernández-Narro, Carlos Sáez
TL;DR
Sepsis fluid and vasopressor dosing requires reliable evaluation of offline reinforcement-learning policies because patient-level prospective trials are unavailable and off-policy estimates can be fragile. This study combines dual off-policy evaluation with reliability and clinician-agreement diagnostics, finding that the learned policy was favored by both estimators while remaining close to observed care.
Problem
Reliable evaluation of sepsis treatment policies remains difficult because off-policy estimates can be optimistic and policies departing from observed care have fewer corroborating trajectories.
Method
The study models sepsis dosing as a discretized Markov decision process and evaluates the learned policy with weighted importance sampling, fitted Q evaluation, effective sample size, and clinician agreement.
Results
WIS 50.8 and FQE 46.8 exceeded clinicians’ return of 38.2, with ESS = 50.1 and total variation 0.18 from observed practice.
Takeaways & Limitations
The learned policy is best interpreted as a clinically plausible refinement of observed care rather than evidence of substantially superior performance.
Takeaways & Limitations
The retrospective off-policy evaluation supports plausibility but does not demonstrate clinical benefit through prospective validation on patients.
Abstract
from arXiv · showhide
The dosing of intravenous fluids and vasopressors in sepsis is a sequential decision made under uncertainty and guided largely by clinical judgment, which makes it a natural target for reinforcement learning from historical care. Because a learned policy cannot be trialed on patients, its value must be estimated off-policy, and such estimates can be fragile and optimistic. This work advances the reliable evaluation of sepsis treatment policies by combining off-policy estimation, reliability diagnostics, and clinician-agreement analyses in a transparent validation framework. We modeled fluid and vasopressor dosing on a cohort of 36,872 septic ICU stays drawn from the MIMIC-IV critical-care database, as a discretized Markov decision process with 1,000 states and 25 actions, defined by a five-by-five grid of fluid and vasopressor levels and solved by policy iteration. The clinicians' behavior policy was estimated with a random forest, which mitigated the collapse of the Effective Sample Size (ESS 50.1 against 4.0 with smoothed counts) that otherwise destabilizes the importance-sampling estimate. The learned policy was evaluated with two estimators, weighted importance sampling (WIS) and fitted Q evaluation (FQE), with the ESS and clinician agreement as reliability checks. An empirical variable selection found that the composition of the state matters more than its size. Both estimators place the learned policy above the clinicians' return (WIS 50.8 and FQE 46.8 against 38.2, ESS 50.1), yet it departs only modestly from observed practice (total variation 0.18), favoring less intravenous fluid. These retrospective single-center off-policy results support the learned policy as a clinically plausible refinement of observed practice and motivate its further evaluation as a discordance-based clinical decision-support approach.
1. Introduction
The introduction frames sepsis treatment as sequential decision-making under uncertainty that must be learned and evaluated offline from retrospective clinical trajectories. It motivates an updated MIMIC-IV pipeline using dual off-policy estimators, reliability diagnostics, and clinician agreement to assess a clinically plausible policy refinement.
- Motivation: 49 million sepsis cases and 13 million related deaths occur annually, with survivors facing physical, cognitive, and mental impairments known as post-sepsis syndrome.The Sepsis-3 definition describes sepsis as life-threatening organ dysfunction caused by a dysregulated host response to infection.
- Problem formulation: Fluid and vasopressor dosing is a sequential decision under uncertainty because repeated actions change the patient’s state and shape subsequent decisions.The formulation uses patient physiology as the state and fluid and vasopressor doses as actions.
- Evaluation challenge: Offline evaluation is necessary because a learned policy cannot be directly trialed on patients, while retrospective off-policy estimates are fragile and potentially optimistic.The introduction identifies evaluation as the central difficulty shared with offline reinforcement learning in healthcare.
- Methodological contributions: The updated MIMIC-IV pipeline uses empirical variable selection and finds that state composition matters more than state size, while removing leakage and ad hoc transformations.All fitted quantities are estimated on the training split alone; fluids are not up-weighted and vasopressors receive no bespoke transform.
- Methodological contributions: Dual WIS and FQE evaluation combines estimators with differing failure modes, ESS reliability diagnostics, and clinician agreement as an independent check.This replaces the single estimator used in the reference work.
- Reliability and findings: 50.1 ESS versus 4.0 with smoothed empirical counts shows that the random-forest behavior-policy estimator controls importance-sampling collapse.The introduction also describes analyzing the ESS, WIS, and FQE trade-offs as the number of states K changes, with K = 1000 treated as a compromise.
2. Background and related work
Sepsis is defined as life-threatening organ dysfunction from a dysregulated response to infection, operationalized by an acute SOFA increase of at least two points. Prior work framed fluid and vasopressor dosing as sequential decisions, while highlighting the trust and transportability challenges of retrospective off-policy evaluation.
- Clinical background: Sepsis-3 defines sepsis as life-threatening organ dysfunction caused by a dysregulated host response to infection, operationalized as an acute SOFA increase of at least two points.SOFA grades respiratory, coagulation, hepatic, cardiovascular, neurologic, and renal function from 0 to 4 each, producing a composite score from 0 to 24.
- Related work: The AI Clinician modeled fluid and vasopressor dosing as sequential decisions over an ICU stay and used a Markov decision process solved by policy iteration.Its policy was designed to maximize estimated survival.
- Related work: The AI Clinician achieved higher estimated value than clinicians, with lower mortality observed when administered doses were closer to recommendations, but its evidence was retrospective rather than trial-based.Its recommendations characteristically favored less intravenous fluid and more low-dose vasopressor.
- Off-policy evaluation: Off-policy evaluation is required for learned policies because they are not executed in observational data and must correct for mismatch with clinicians’ behavior policy.Clinicians’ return can be estimated directly from observed outcomes, whereas learned-policy evaluation requires corrective estimators such as importance sampling.
- Limitations: Observational off-policy reinforcement learning is difficult to trust because omitted variables can confound associations and violate the Markov assumption.Policies that depart from observed practice have fewer corroborating trajectories, making potentially valuable policies harder to evaluate.
- Limitations: Retrospectively validated policies may fail to transfer prospectively because of distribution shifts across sites and time and because sparse terminal rewards can poorly proxy clinical benefit.These limitations further constrain interpretation of retrospective policy evaluation.
3. Data and cohort
The study used MIMIC-IV to construct a filtered adult Sepsis-3 cohort of 36,872 ICU stays, with patient-level partitioning and outcomes defined from complementary mortality indicators. The cohort was demographically and clinically characterized by age, sex, severity, vital signs, and organ-damage markers.
- Cohort construction: 36,872 ICU stays remained after Sepsis-3 identification and exclusion of treatment withdrawal and stays without documented intravenous fluid.The cohort began with 94,458 ICU stays; 41,295 met the adult Sepsis-3 definition, while withdrawal and no-fluid filters removed 573 and 3,850 stays, respectively.
- Cohort construction: 28,605 unique patients contributed the cohort, and the 80/20 training-validation split was drawn by patient to prevent trajectory leakage.Data were extracted from onset − 24 h to onset + 48 h, with the effective MDP trajectory spanning onset to onset + 48 h.
- Cohort characteristics: Median age was 66 years (interquartile range 55 to 76), and 58.0 % of the cohort was male.The cohort included roughly twice as many stays as the referenced AI Clinician cohort while showing similar demographic composition.
- Cohort characteristics: Mean SOFA at onset was 7.7 ± 2.6, while median heart rate, mean arterial pressure, oxygen saturation, and Glasgow Coma Scale were 86 bpm, 77 mmHg, 98 %, and 15.Lactate was elevated at the median (2.1 mmol L−1), and creatinine reached 1.9 mg dL−1 at the third quartile; SOFA was recomputed on imputed data.
- Outcome definitions: 100 % of admissions had an in-hospital mortality flag defining terminal reward, while deathtime truncated trajectories and date of death supported 90-day mortality.The three indicators had different coverage and roles in defining MDP outcomes and censoring.
4. Methods
The study formulates sepsis hemodynamic management as a discretized MDP built from SOFA-related clinical state features and 4-hour patient windows. Policy iteration uses support-based action masking, while model selection prioritizes reliable, clinically plausible policies supported by both off-policy estimators.
- State representation: State features covered six SOFA organ systems, vital signs, lactate, and slowly varying context including weight, ventilation, comorbidity, age, and sex.Variables were extracted from MIMIC-IV hosp and icu modules.
- MDP construction: Each 4 h window was assigned to the nearest of K MiniBatch k-means clusters, with hospital discharge and death appended as absorbing states, giving K + 2 states.Clustering excluded outcome columns and was fit only on the training split.
- MDP construction: The action space contained 5 × 5 = 25 combinations of discretized intravenous fluid and vasopressor levels.Fluids were summed within each window, whereas vasopressor exposure used the maximum norepinephrine-equivalent dose; dose cutoffs were derived from positive training-split values.
- Policy optimization: M = 25 observations was the support threshold for masking unreliable state-action pairs before policy improvement.Actions with insufficient support or empty transition rows received Q(s, a) = −∞; states with every action masked used the clinicians’ modal action, and the resulting policy was softened with ε = 0.01.
- Model selection: ESS below 50 triggered the reliability gate, followed by clinical-plausibility and WIS–FQE triangulation gates during model selection.The procedure favored policies with non-intervention rates close to clinicians’ practice and with both WIS and FQE above clinicians’ return.
5. Results
Results identified a clinically grounded state representation and selected a robust final policy configuration through plausibility and dual-evaluation filters. The learned policy exceeded clinicians’ estimated return, retained substantial agreement with observed care, and showed lower mortality where doses coincided with recommendations.
- Variable selection: Composition dominated state size: curated variables achieved roughly 17.5% intervention agreement versus near 7% for equal-sized random sets, with fluid volume explaining 56% of intervention-rate variance.Adding was_measured indicators did not improve agreement, with mean paired differences from −0.17 to −1.15 points.
- Configuration selection: Only 3 configurations passed the strict requirement that both WIS and FQE lower confidence bounds exceed clinicians’ return, all at K = 1000.The sweep evaluated 1200 policies; 65 survived the plausibility gate and 26 survived the point-estimate return criterion.
- Dual off-policy evaluation: Clinicians’ return was 38.2 [37.2, 39.1], compared with 50.8 [41.2, 58.6] for WIS and a higher learned-policy estimate from FQE.The two estimators agreed in placing the learned policy above clinicians; the final configuration used a lower discount, so returns are not comparable across discounts.
- Clinician agreement: 44.4% of recommended actions matched clinicians overall, rising to 69.4% when a one-level dose deviation counted as agreement.Agreement was 79.0% for vasopressors and 52.0% for fluids.
- Dose divergence and mortality: Observed mortality was lowest when clinician doses coincided with learned recommendations and rose steadily as fluid or vasopressor actions diverged.This descriptive relationship was assessed separately for fluids and vasopressors on validation steps.
6. Discussion
The discussion presents the learned policy as a transparently and cautiously evaluated refinement of observed sepsis practice, not evidence of dramatic superiority. Its clinical plausibility supports future discordance-based decision support, but retrospective single-center data do not establish prospective benefit or transferability.
- Contribution: The work’s main contribution is a trustworthy, disciplined account of policy evaluation rather than an unqualified claim that the learned policy is better.It replaces leakage-prone preprocessing with empirical variable selection and training-split-only estimation, and pairs WIS with FQE instead of relying on one estimator.
- Policy interpretation: 0.18 total variation shows that the learned policy is a refinement of observed practice rather than a qualitatively new strategy.It withholds treatment in 42.8% of steps, far below the 80%–85% degenerate-policy range, and characteristically recommends less intravenous fluid.
- Clinician agreement: Agreement is higher when clinicians withhold treatment than when they administer it, indicating that policy departures concentrate in harder treatment cases.This asymmetry also reflects the agreement–intervention circularity identified as a limitation.
- Evaluation limits: 50.1 effective sample size barely clears the preset minimum of 50, while reliable configurations consistently use γ = 0.95, identifying important limits to the evaluation.The off-policy tradeoff remains central: more divergent policies may offer greater value but have less data to corroborate them.
- Clinical plausibility: Observed mortality is lowest at exact clinician–policy agreement and rises with dose divergence for both fluids and vasopressors, but this association is not causal evidence.The discussion also notes that the cohort resembles an externally validated reference population, while single-center training leaves distribution shift and prospective transfer unresolved.
- Future use: The proposed future role is decision support that highlights discordant dosing without replacing clinical judgment, while prospective validation remains necessary to demonstrate clinical benefit.Retrospective off-policy evaluation supports plausibility and consistency, not demonstrated patient benefit.
7. Limitations and future work
The study is limited by state–action circularity, approximate state representation, fragile tabular off-policy evaluation, sparse reward design, and single-center distribution shift. Future work prioritizes lagged doses, richer rewards, additional estimators, and prospective clinician-in-the-loop decision support.
- Limitations: Current-step doses enter both state and action, making the query state non-prospectively constructible and creating direct state–action confounding.The formulation also relies on a Markov approximation to a partially observable problem.
- Limitations: K-means assigns poorly represented patients to the nearest learned state, while the Markov and tabular assumptions discard relevant clinical structure.After support filtering, 81.7 % of state-action pairs lack a reliable transition, 3.1 % of states use the clinicians’ mode, and 3.2 % are very small clusters.
- Limitations: ESS = 50.1 barely exceeds the preset floor of 50, and only 77 of 1200 configurations proved reliable, limiting confidence in off-policy evaluation.Greater policy departure from clinicians increases potential value but reduces available corroborating data.
- Limitations: The terminal binary mortality reward omits intermediate objectives and later quality of life, while single-center training creates risk of distribution shift.The paper identifies richer intermediate signals and external prospective validation as open needs.
- Future work: Lagging doses to t −1 would break state–action circularity, while denser lactate or SOFA signals could provide a richer reward than final outcome alone.Automatic variable selection is proposed only as a complement to clinical judgment because the best results restricted variables to clinically essential features.
- Future work: A clinician-in-the-loop decision-support tool could flag dosing discordance, provide prospective validation, support updating, and help address distribution shift.Additional refinements include weighted doubly robust estimation, alternative behavior-policy models, and alternatives to k-means.
8. Conclusions
The study’s main contribution is a disciplined dual off-policy evaluation of an offline sepsis treatment policy, combining WIS, FQE, ESS reliability diagnostics, and clinician agreement. Results support further clinical validation, but retrospective single-center evidence limits conclusions.
- Evaluation framework: Dual off-policy evaluation paired weighted importance sampling with fitted Q evaluation, using effective sample size and clinician agreement as reliability checks.This extends the reference work’s single-estimator evaluation with an independent clinical plausibility check.
- Key findings: State composition mattered more than variable-set size: clinically essential variables outperformed larger sets that tended to degenerate toward inaction.The conclusion identifies the signal-to-noise ratio of the state as dependent on the selected variables’ composition.
- Key findings: WIS 50.8 and FQE 46.8 exceeded the clinicians’ return of 38.2, with ESS = 50.1 above the prespecified reliability floor.Both estimators placed the learned policy above clinicians’ return while meeting the stated ESS reliability criterion.
- Limitations and next steps: Retrospective, single-center, off-policy evidence supports further policy validation in clinical decision support rather than definitive clinical adoption.Recommended next steps are external validation, temporally cleaner state definitions, and prospective assessment.
Supplementary Material
The supplementary material accompanies the study “Offline Reinforcement Learning for Hemodynamic Management of Sepsis in the ICU: a MIMIC-IV Study with Dual Off-Policy Evaluation.” It collects descriptive tables and secondary figures cited from the main text in order of appearance.
- The supplementary material concerns offline reinforcement learning for hemodynamic management of sepsis in the ICU.
- The study is identified as a MIMIC-IV study with dual off-policy evaluation.
- The material collects descriptive tables and secondary figures referenced from the main text, cited in order of appearance.
1 Supplementary figures
The supplementary figures document cohort selection and the preprocessing pipeline used to construct the sepsis MDP. The final cohort included 36 872 ICU stays, while preprocessing comprised extraction, 4 h discretization, imputation, scaling, clustering, and model estimation.
- Cohort selection: 36 872 ICU stays (28 605 unique patients) formed the final cohort, representing 89.3% of the 41 295 adult Sepsis-3 stays after treatment-withdrawal and undocumented-fluid exclusions.The cohort was selected from 94 458 MIMIC-IV ICU stays.
- MDP construction: The preprocessing and MDP-construction pipeline included extraction, 4 h-window discretization, imputation, scaling, state clustering, and estimation of transition and reward models.These steps defined the data-processing workflow before policy evaluation.
2 Supplementary tables
The supplementary tables document the cohort’s observed baseline state, the 22-variable final configuration, five-level action discretization, and candidate variable sets used in the final sweep. They also highlight incomplete onset laboratory coverage and the contrast between curated and control variable sets.
- Baseline cohort state: Table S1 summarizes the cohort’s initial 4 h clinical state using observed values without imputation.Several laboratory variables at onset have incomplete coverage, motivating the main-text imputation strategy.
- State configuration: 22 variables comprise the final state configuration, with variables transformed by logarithmic transformation, standardization, or centered binarization before clustering.The empirical selection of this configuration is described in the main text.
- Action discretization: Five action levels are assigned to each treatment lever using quartiles of strictly positive training-split doses.Vasopressor doses are expressed as norepinephrine equivalents.
- Candidate variable sets: Five candidate variable sets enter the final sweep, with full-set and Komorowski-replica controls reproducing non-intervention collapse while curated sets do not.This comparison distinguishes the control configurations from the curated alternatives.