Source-linked AI summary
Outlier detection for patient monitoring and alerting
Miloš Hauskrecht, Iyad Batal, Michal Valko, Shyam Visweswaran, Gregory F. Cooper, Gilles Clermont
TL;DR
Knowledge-based alerting can be time-consuming to build and clinically limited, motivating data-driven ways to identify potential management errors. This paper models unusual patient-management actions from past EHR cases and evaluates generated alerts with expert judgment. The evaluation found true alert rates from 0.25 for weak alerts to 0.66 for stronger alerts.
Problem
Expert-built alerting systems can be time-consuming to develop and have limited clinical coverage, while unusual patient-management actions may indicate errors.
Source: §1 · p. 1Method
The approach uses past EHR cases to learn conditional outlier models that detect patient-management actions unusual for a patient's condition.
Source: §1 · p. 1, §3.1.1 · p. 3Results
0.25 true alert rate for weak alerts, rising to 0.66 for stronger alerts.
Source: §6 · p. 8Takeaways & Limitations
Expert judgment supports the approach as a data-driven complement to rule-based alerting for identifying potential medical errors.
Source: §7 · p. 8The current action-specific models omit multivariate relations among actions and use 24-hour patient-record segmentation.
Source: §7 · p. 8
Abstract
from arXiv · showhide
We develop and evaluate a data-driven approach for detecting unusual (anomalous) patient-management decisions using past patient cases stored in electronic health records (EHRs). Our hypothesis is that a patient-management decision that is unusual with respect to past patient care may be due to an error and that it is worthwhile to generate an alert if such a decision is encountered. We evaluate this hypothesis using data obtained from EHRs of 4486 post-cardiac surgical patients and a subset of 222 alerts generated from the data. We base the evaluation on the opinions of a panel of experts. The results of the study support our hypothesis that the outlier-based alerting can lead to promising true alert rates. We observed true alert rates that ranged from 25\% to 66\% for a variety of patient-management actions, with 66\% corresponding to the strongest outliers.
1. Introduction
Expert-built monitoring systems can be time-consuming to develop and have limited clinical coverage. This paper proposes learning from past EHR cases to identify unusual patient-management actions as potential errors and evaluates whether such outliers correspond to clinically useful alerts.
1. Introduction
- The proposed data-driven framework learns from stored clinical information in past patient cases to detect clinical outliers.The paper expands on methods and results originally reported in.Source: §1 · p. 1
- Outlier-based monitoring may complement knowledge-based alerts to broaden clinical coverage, or work alone where knowledge-based alerting is unavailable.Knowledge-based systems depend on expertise extracted in advance and are time-consuming to build, with limited clinical coverage.Source: §1 · p. 1
- The approach targets actions unusually made for patients with similar conditions, rather than unusual patient data instances in general.A detected deviation may generate a patient-specific alert for clinicians to consider.Source: §1 · pp. 1–2
- Outlier detection was positively correlated with clinically meaningful alerts in 4486 post-cardiac surgical patients, supporting the hypothesis that unusual care decisions can yield useful alerts.A panel of 15 critical-care experts evaluated a subset of 222 generated alerts.Source: §1 · p. 2
2. Background
Knowledge-based alert systems encode expert clinical knowledge as rules and cover many decision-support tasks, but their development and performance have practical constraints. The paper presents care-outlier detection as a data-driven complementary approach.
2. Background
- Rule-based systems can require tedious expert input, cover only a limited part of potential adverse events, and be difficult to tune; false alerts can lead to rules being retired or ignored.These systems have been applied to medication decision support and other clinical tasks.Source: §2 · p. 2
- The proposed approach uses conditional outlier methods to find patient-management actions unusual among past patients with the same or similar conditions.The paper presents outliers as actions that may indicate management errors.Source: §2 · p. 2
- The authors describe the approach as requiring no expert input to build, drawing on prior patient cases, and potentially offering broad and deep alert coverage.They characterize the approach as having significant potential for positive impact on clinical care.Source: §2 · p. 2
3. Methods
The methods learn action-specific models from past EHR cases, then apply them to patient states to flag care decisions that are unusual under the learned patterns. Patient records are segmented and represented as features, predictive models estimate action probabilities, and an alert score combines anomaly signals across successive states.
3. Methods
- The framework has model-building and model-application stages: past EHR cases train action-specific outlier models, which are applied to new patient data to find actions that deviate from prevalent care patterns.Models can capture patient subpopulations in which an action is typical, atypical, or discontinued.Source: §3 · p. 2
3.1. Conditional anomaly models
- Conditional anomaly detection identifies unusual outcomes or actions for a subset of attributes given the remaining context, here patient-management actions given patient condition.This differs from detecting unusual data instances without conditioning on patient state.Source: §3.1 · pp. 2–3
3.1.1. Measure of deviation
- An action is anomalous when its conditional probability given the patient state is small; its anomaly score is 1 − P(y|x).Predictive models for each action are built from past patient data and applied to observed state-action pairs.Source: §3.1.1 · p. 3
3.1.2. Segmentation of the EHR data
- EHR time series are segmented at discrete points into patient states linked to the management actions occurring before the next segmentation point.The example uses 24-hour intervals; each patient-state instance can cover a different hospitalization period.Source: §3.1.2 · p. 3
3.1.3. Representation of patient state and management actions
- Patient states are converted to fixed-length feature vectors, including laboratory time-series summaries and medication and procedure features.Continuous or ordinal laboratory values use 28 features, including changes, slopes, nadirs, and apexes; medication and procedure features include indicators and elapsed times.Source: §3.1.3 · p. 3
3.1.4. Learning predictive probabilistic models
- For each action, the method learns P(y|x) with an SVM and greedily adds feature groups when they improve AUC, considering at most 15 groups.A non-parametric estimate derived from SVM projections is used to estimate class posterior probabilities.Source: §3.1.4 · pp. 3–4
3.2. The generation of alerts
- The alert score takes the minimum of anomaly scores for the same action under the previous and current patient states, so an alert requires the unusual signal to persist across both time steps.An alert is generated when this score exceeds a threshold.Source: §3.2 · p. 4
4. Evaluation of the outlier-based alerting methodology
The evaluation applied anomaly models to a time-split cohort of post-surgical cardiac patients, selected alerts using action-specific thresholds, and assessed their usefulness through physician review.
4.1. Dataset
- The study divided 4,486 post-surgical cardiac patient records into 2,878 training cases and 1,608 test cases.Daily segmentation produced 51,492 patient-state instances: 30,828 for training and 20,664 for evaluation.Source: §4.1 · p. 5
4.2. Selection of models for evaluation
- Only models with AUC of 0.68 or higher were retained for detecting lab omissions, medication omissions, and medication continuations.The models used SVMs with greedy group-based feature selection.Source: §4.2 · p. 5
4.3. Selection of alerts for evaluation
- Action-specific anomaly and alert-score thresholds filtered the test-set candidates, yielding 4,870 candidates and 222 alerts on 100 patients for review.The 222-alert sample favored higher alert scores and patients with multiple alerts; Fig. 5 shows score distributions for the evaluated alerts and initial candidate pool.Source: §4.3 · p. 5, Figure 5 · p. 6
4.4. Review of alerts
- Fifteen post-cardiac-care physicians reviewed the alerts, and an alert counted as useful when at least two of its three reviewers judged it useful.Reviewers also assessed whether they would follow up with a patient-management action; that outcome likewise used a two-of-three majority rule.Source: §4.4 · pp. 5–6
5. Results
Reviewer agreement was summarized across five groups, while true-alert analyses covered all evaluated alerts, stronger-score alerts, and the relationship between alert scores and reviewer assessments. Illustrative cases included both correct and incorrect recommendations.
5.1. Reviewer agreement
- Table 2 reports minimum and maximum pairwise reviewer agreement and Cohen’s kappa across the five study groups.The passage describes the measures but does not provide their numerical values.Source: Table 2 · p. 6, §5.1 · p. 6
5.2. True alert rates
- Tables 3 and 4 report true alert rates and 95% confidence intervals for all evaluated alerts and for the alert-score 0.8–1.0 subset.Table 3 also summarizes results by alert category.Source: Table 3 · p. 6, §5.2 · pp. 6–7, Table 4 · p. 7
5.3. Alert scores versus true alert rates
- True alert rates rose from 25% to 66% for Item 1 and from 25% to 64% for Item 2 as alert scores increased.Alert-score AUC was 0.64 for Item 1 and 0.63 for Item 2; both were statistically significantly different from 0.5 (p-value 0.05).Source: Figure 6 · p. 7, §5.3 · p. 7
5.4. Illustrative examples of alerts
- Examples included levothyroxine and potassium omission alerts assessed as correct, alongside heparin and warfarin recommendations assessed as incorrect.The heparin alert lacked progress-note context about repeat surgery and bleeding; the warfarin alert did not account for the mechanical-valve INR context.Source: §5.4 · p. 8
6. Discussion
The evaluation found true alert rates from weak to strong outliers that compared favorably with published false-alert rates under an explicit assumption. The authors also identify deployment-evaluation caveats and model limitations, while suggesting alert scores could help tune performance.
6. Discussion
- True alert rates ranged from 0.25 for weak alerts to 0.66 for stronger alerts, comparing favorably with published false-alert rates under an override-rate assumption.The comparison assumes literature override rates closely approximate false-alert rates.Source: §6 · p. 8
- True alert rates were positively correlated with alert scores, suggesting adjustment toward desired true alert rates may be possible.The proposed control direction uses empirically derived measures rather than annotated alert data.Source: §6 · p. 8
- Offline retrospective evaluation omitted deployment factors such as alert fatigue and workflow interruption, limiting direct inference about deployed performance.Source: §6 · p. 8
- Incorrect alerts reflected text-only information, medication interactions, lab-panel correlations, uncommon alternatives, and missed temporal care patterns; model and EHR improvements may raise true alert rates.Source: §6 · p. 8
7. Conclusions
Expert judgment supported clinically valid outlier alerts across a broad range of conditions and patient-management actions, using models built from EHR data without expert-built detection rules. The evaluation and current models remain constrained by retrospective review and modeling choices.
7. Conclusions
- Expert judgment found clinically valid outliers at reasonably high true positive alert rates across broad clinical conditions and actions, using EHR data without expert-built detection models.Source: §7 · p. 8
- Offline retrospective review omitted workflow interruption and alert fatigue, while reviewer capacity restricted how many aspects of the system could be studied.Source: §7 · p. 8
- Per-action models omit multivariate relations, 24-h segmentation may impair short-horizon prediction, and current features failed to yield good models for some actions.Source: §7 · p. 8
- More suitable features for complex time-series data may improve models and broaden coverage of patient-management actions.Source: §7 · p. 9