Removing Temporal Note Redundancy Improves Multimodal Reinforcement Learning for Medicine

Chenran Weng; Joo Seung Lee; Malini Mahendra; Anil Aswani

Source-linked AI summary

Removing Temporal Note Redundancy Improves Multimodal Reinforcement Learning for Medicine

Chenran Weng, Joo Seung Lee, Malini Mahendra, Anil Aswani

arXiv:2608.14157v1Aug 14, 2026cs.AIcs.LG

TL;DR

Mechanical-ventilation RL states often rely on structured EHR data while longitudinal notes contain redundant text that can obscure new clinical information. This paper removes temporal note redundancy with embedding-space and sentence-level methods, and both representations improve estimated policy value over structured-only and raw-note baselines across multiple off-policy evaluation methods.

Problem
Mechanical-ventilation RL state representations mainly use structured EHR variables, while redundant longitudinal notes can obscure decision-relevant, time-local clinical information.
Method
The paper constructs redundancy-aware multimodal offline-RL states by removing duplicated note content using embedding-space decomposition and sentence-level differencing.
Results
Across multiple off-policy estimators, both redundancy-aware representations improve estimated policy value over structured-only and raw-note baselines; sentence-level differencing gains +0.27 under FQE over structured-only states.
Takeaways & Limitations
Explicitly modeling temporal redundancy makes clinical notes useful additional information for offline RL and supports more effective multimodal states for sequential decision-making.
Takeaways & Limitations
The retrospective offline-RL evaluation provides no direct evidence of clinical benefit, and results come from a single ICU dataset, MIMIC-III.

Abstract

from arXiv · show

Mechanical ventilation is a critical life-support intervention, requiring dynamic adjustments to ventilator settings as a patient's condition evolves. While reinforcement learning (RL) offers a promising framework for optimizing these sequential decisions, standard approaches rely primarily on structured electronic health record (EHR) data, missing crucial clinical context recorded in free-text notes. Integrating longitudinal clinical notes into RL state spaces is challenging because notes are heavily inflated by temporal redundancy, such as copy-forward text, templating, and repetitive documentation, which dilutes time-local updates and degrades state representation quality. To address this, we propose a redundancy-aware multimodal state representation framework that explicitly removes duplicated note text over time before policy learning. We evaluate two computationally efficient temporal decomposition strategies for removing duplicated note text: (1) an embedding-space decomposition using singular value decomposition on local history subspaces, and (2) an interpretable sentence-level diff operation that filters out previously documented sentences before text encoding. Using real-world ICU data, we demonstrate that state representations constructed by stripping temporal note redundancy significantly outperform both structured-only and raw-note baselines across multiple off-policy evaluation methods (Model-Based Rollouts, Fitted Q-Evaluation, Weighted Importance Sampling, and Weighted Doubly Robust Evaluation). Our findings show that explicitly isolating new clinical information from repeated note text yields higher-quality state representations and directly improves RL performance for clinical decision support.

Introduction

Mechanical ventilation requires continuously adapting settings, yet structured EHR-based reinforcement learning can miss clinical context in longitudinal notes. This work removes temporal note redundancy with efficient decomposition methods and evaluates redundancy-aware multimodal states for offline reinforcement learning.

Clinical background: Mechanical ventilation requires continuous ventilator-setting adjustments as patient conditions evolve, while inappropriate settings can cause serious complications.Reported complications include ventilator-induced lung injury, diaphragm dysfunction, pneumonia, and oxygen toxicity.
Motivation: Prior reinforcement-learning approaches for mechanical ventilation mainly use structured EHR variables, which may omit clinical context captured in notes.Unlike supervised learning, reinforcement learning optimizes long-term outcomes from sequential states, actions, and rewards.
Documentation challenge: Longitudinal clinical notes contain valuable assessments and treatment plans but are temporally redundant because of templating, repeated documentation, copy-paste, and copy-forward practices.A study of approximately 2.3 million inpatient notes found that 42% used copy-paste.
Contributions: The paper introduces two computationally efficient decomposition methods that separate history-related content from newly introduced information without repeated large-language-model processing.The methods operate in embedding space and at the sentence level.
Evaluation: On MIMIC-III mechanical-ventilation trajectories, redundancy-aware multimodal state representations improve estimated policy value over structured-only and raw-note multimodal baselines across multiple off-policy evaluation methods.The evaluation methods include Model-Based Rollouts, Fitted Q-Evaluation, Weighted Importance Sampling, and Weighted Doubly Robust Evaluation.

Related Work

Prior work has studied reinforcement learning for mechanical ventilation and multimodal EHR learning, while separate research has characterized the redundancy introduced by repeated clinical-note documentation. These strands motivate combining dynamic ventilator control with note-based representations that account for redundancy.

Reinforcement learning for mechanical ventilation: Prior reinforcement-learning studies addressed mechanical-ventilation optimization through dynamic control, model-based policy learning, interpretability, safety, and weaning.VentAI used tabular Q-learning with structured patient trajectories to recommend ventilation regimes and evaluate their association with mortality.
Multimodal EHR learning: Multimodal EHR research shows that clinical notes can complement structured variables for medical prediction across fused feature types and ICU outcome prediction.Reported gains from notes may depend strongly on their clinical information rather than only on model architecture.
Clinical-note redundancy: Clinical notes contain repeated information from templates, copy-paste, and copy-forward practices, motivating studies of redundancy, its sources, and tools for monitoring copy-paste use.Prior work examined redundancy’s effects on text-mining performance and described where duplicate information in electronic medical records originates.

Method

The method formulates mechanical-ventilation management as finite-horizon offline RL with fixed 4-hour clinical decisions, then constructs multimodal states by separating note history from newly introduced information. It compares embedding-space residual decomposition with an interpretable sentence-level diff before fusing note features with structured variables.

RL formulation: Mechanical-ventilation management is modeled as finite-horizon offline RL using 4-hour decision intervals and fixed clinical variables, action discretization, and reward design.Patient trajectories are represented as transitions (s_t, a_t, r_t, s_t+1).
State construction: The structured state contains respiratory, hemodynamic, blood-gas, and miscellaneous variables, augmented with a time-invariant propensity score, and serves as the baseline representation.Clinical-note representations are added to this structured baseline for comparison.
Action space: The action is a discretized ventilator-setting tuple, with clinically meaningful bins producing a finite discrete action space for offline discrete RL.The tuple includes ideal-bodyweight-adjusted tidal volume, positive end-expiratory pressure, and fraction of inspired oxygen.
Note processing: Clinical notes are restricted to information available by each 4-hour state-construction cutoff, while redundancy-aware representations separate persistent history from time-local innovation.The motivation is that copy-forward, templating, and repeated historical content can obscure newly introduced clinical information relevant to control.
Redundancy-aware representations: Embedding-space decomposition projects the current note embedding onto a patient-local recent-history subspace and treats the residual as innovation, whereas sentence-level diff explicitly splits historical and newly added sentences before separate encoding.The final RL state concatenates structured variables with compressed note-derived history and new-information features.
Sentence-level diff: The sentence-level diff removes redundancy earlier and more explicitly: its history channel contains text traceable to previous notes, while its new channel contains text introduced at the current time.This design is particularly natural when duplication occurs through sentence-level copy-forward.

Experiments · Dataset

The study evaluates mechanically ventilated adult ICU trajectories from MIMIC-III, combining structured observations with temporally aligned clinical notes. Episodes are discretized into four-hour decisions, truncated to 72 hours, and supported by 157,924 notes from 10,125 ICU stays.

Dataset: The dataset comes from MIMIC-III, an openly accessible critical care database, and follows a previously described structured-data preprocessing pipeline.The cohort focuses on ICU stays involving mechanical ventilation.
Dataset: Patient trajectories are discretized into four-hour intervals, with each episode containing demographic, laboratory, vital-sign, and ventilator-related variables.These structured observations define the decision-time framework for the cohort.
Dataset: Continuous structured variables are winsorized to reduce extreme-outlier effects, while missing values are handled through forward/backward filling and KNN imputation.The preprocessing combines multiple missing-data strategies rather than relying on a single imputation method.
Dataset: Trajectories are truncated to the first 72 hours of ventilation, producing a maximum of 18 decision points per episode.There is one decision point every four hours.
Dataset: Clinical notes are aligned with the same four-hour intervals as structured observations and concatenated within each interval.De-identified note time and note category are preserved as headings.
Dataset: The final note dataset contains 157,924 notes from 10,125 ICU stays aligned to 147,997 decision intervals.Each ICU stay contains 15.60 notes on average.

Off-Policy Evaluation

The study evaluates each learned target policy with four complementary off-policy estimators to triangulate policy value. These methods differ in their estimation mechanisms and bias–variance tradeoffs, with model-based rollouts additionally sensitive to transition-model accuracy and training support.

Evaluation framework: Four complementary estimators evaluate each target policy: model-based rollouts, Fitted Q Evaluation, Weighted Importance Sampling, and Weighted Doubly Robust evaluation.Reporting all four methods supports triangulation of policy value despite different bias–variance tradeoffs.
Model-Based Rollouts: Model-based rollouts simulate evaluation-policy trajectories using a learned probabilistic transition model and discounted rewards from test-set initial states.The method averages 5 independent trajectories per initial state to reduce stochastic-rollout variance.
Model-Based Rollouts: Model-based rollouts are unbiased only when the learned transition model matches the true kernel and may be optimistic in sparsely supported state–action regions.This limitation arises when the evaluation policy visits regions with limited training data.
Fitted Q Evaluation: Fitted Q Evaluation estimates policy value by iteratively fitting a Q-function under the evaluation policy.The final value is computed from the learned Q-function.
Importance-Based Estimators: Weighted Importance Sampling reweights observed trajectories using evaluation-to-behavior policy ratios, while Weighted Doubly Robust combines this correction with FQE estimates to reduce variance.The behavior policy is estimated with behavior cloning plus a small uniform mixture, ϵ = 0.05.

Implementation Details

Implementation used an 80:20 episode split, five-fold training-set cross-validation, and model-based validation with a safety-constrained objective. Final CQL and behavior-cloning models used fixed hyperparameters after cross-validation.

Data splitting and tuning: Mechanical ventilation episodes were split into training and testing subsets in an 80:20 ratio, with 5-fold cross-validation on training data.Cross-validation selected shared reward and policy-learning hyperparameters using the structured-only baseline representation.
Model-based evaluation: Model-based rollouts initialized from observed ICU-stay states, while CQL selected actions and a learned transition model predicted subsequent states.Validation performance was averaged across stays using simulated per-step SpO2 changes.
Transition-model evaluation: 1.080 mean absolute error and 0.903 ROC-AUC characterized next-step SpO2 prediction on the held-out test set.The ROC-AUC measured classification of whether SpO2 exceeded 94%.
Final model configuration: Final models were trained with fixed hyperparameters after cross-validation, including γ = 0.99 for CQL policies.CQL used d3rlpy with a two-layer 256-unit-per-layer MLP and Adam learning rate 6.25×10−5; behavior cloning used [128, 128] hidden units and learning rate 10−4.

Numerical Results

Redundancy-aware state representations consistently improved estimated policy value over structured-only and raw-note baselines across FQE, WIS, WDR, and MB. A held-out ICU trajectory illustrates that they preserve stable ventilation context while separating new clinical information, maintaining clinician-consistent recommendations.

Aggregate OPE Comparisons Across State Representations: Sentence-level diff improved over structured-only by +0.27 (FQE), +0.64 (WIS), +0.41 (WDR), and +0.75 (MB), with all confidence intervals excluding zero.The embedding-space SVD representation also improved by +0.12 (FQE), +0.40 (WIS), +0.22 (WDR), and +0.73 (MB).
Aggregate OPE Comparisons Across State Representations: The SVD representation improved over raw notes by +0.14 to +0.46, while sentence-level diff improved by +0.28 to +0.69, depending on the estimator.Raw notes showed no consistent improvement over structured-only: confidence intervals included zero under FQE, WIS, and WDR, while MB alone had a positive point estimate.
Aggregate OPE Comparisons Across State Representations: Conclusions relied on consistent improvements across all four OPE methods, despite FQE producing narrower bootstrap intervals than WIS and WDR.The passage attributes this difference to value-function estimation versus trajectory-level importance weighting and differing bias–variance tradeoffs.
Held-Out Trajectory Analysis: Redundancy-aware representations preserved stable ventilation context while separating newly introduced information, allowing learned policies to remain consistent with the clinician’s action.At t = 9, notes added broader clinical information while ventilation-related information remained unchanged; raw-note embeddings could instead change recommendations based on less relevant events.

Conclusion

The framework improves offline RL state construction for mechanical ventilation by removing temporal redundancy from longitudinal clinical notes before policy learning. Both redundancy-aware representations improve estimated policy value over structured-only and raw-note multimodal baselines, while retrospective offline-RL limitations constrain interpretation.

Contribution: The study presents a redundancy-aware multimodal state representation framework for offline RL in mechanical ventilation.It targets temporal redundancy that can obscure decision-relevant information in raw note embeddings.
Method: Two computationally efficient decompositions separate history-related from newly introduced clinical information before policy learning.They operate in embedding space using recent note history or at sentence level using explicit repeated-versus-new text differences.
Results: Both redundancy-aware representations improve estimated policy value over structured-only and raw-note multimodal baselines across multiple off-policy evaluation methods.The evaluation uses MIMIC-III mechanical ventilation trajectories.
Limitations: Retrospective offline-RL findings do not directly establish clinical benefit and remain vulnerable to dataset bias, distribution shift, and limited state-action coverage.The decomposition methods may identify textual or representational changes that do not always correspond to clinically meaningful updates.