PaperlayerView exact revision on arXiv

Source-linked AI summary

ClinicalVis: Supporting Clinical Task-Focused Design Evaluation

Marzyeh Ghassemi, Mahima Pushkarna, James Wexler, Jesse Johnson, Paul Varghese

arXiv:1810.05798v1cs.HC

TL;DR

ClinicalVis addresses the difficulty of using task-agnostic EHRs for specific ICU care-planning decisions. It introduces an open-source visualization prototype and evaluates it with 14 HCPs using real patient data. Accuracy remained generally poor, but ClinicalVis improved the experience of reviewing data, increased confidence, and shifted time toward validating care plans.

  • Problem

    Task-agnostic EHRs are difficult to use for planning specific clinical tasks, and no open-source framework provided task-focused visual summaries with subsequent HCP-response evaluation.

    Source: §1 · p. 1

  • Method

    ClinicalVis is an open-source EHR visualization prototype evaluated against a baseline with 14 HCPs using real patient data in a simulated eICU care-planning study.

    Source: §1 · p. 2, §2 · p. 5

  • Results

    HCP forecasting accuracy was generally poor and did not significantly improve with ClinicalVis, while HCP experience and confidence improved and time shifted toward care-plan validation.

    Source: §1 · p. 2, §5 · p. 7, §7 · p. 10

  • Takeaways & Limitations

    Task-focused EHR evaluation can reveal how visualization changes HCP experience and validation behavior even when it does not improve task performance.

    Source: §1 · p. 2, §5 · p. 7

    The study examined two tasks in one care-planning setting with limited displayed data and a limited HCP sample, requiring broader future evaluation.

    Source: §6 · p. 10

Abstract

from arXiv · show

Making decisions about what clinical tasks to prepare for is multi-factored, and especially challenging in intensive care environments where resources must be balanced with patient needs. Electronic health records (EHRs) are a rich data source, but are task-agnostic and can be difficult to use as summarizations of patient needs for a specific task, such as "could this patient need a ventilator tomorrow?" In this paper, we introduce ClinicalVis, an open-source EHR visualization-based prototype system for task-focused design evaluation of interactions between healthcare providers (HCPs) and EHRs. We situate ClinicalVis in a task-focused proof-of-concept design study targeting these interactions with real patient data. We conduct an empirical study of 14 HCPs, and discuss our findings on usability, accuracy, preference, and confidence in treatment decisions. We also present design implications that our findings suggest for future EHR interfaces, the presentation of clinical data for task-based planning, and evaluating task-focused HCP/EHR interactions in practice.

A Visualization-Based Prototype System to Explore the Interaction of Healthcare

ClinicalVis addresses the lack of open-source, task-focused EHR visualization frameworks by combining a visualization prototype with empirical evaluation using real patient data. The study examines HCP-EHR interaction during clinical care planning and reports implications for future interfaces.

1 INTRODUCTION

  • Commercial EHRs are task-agnostic, cognitively cumbersome, and difficult to evaluate because they are closed-source and clinical access is constrained.Source: §1 · p. 1
  • ClinicalVis provides an open-source visualization prototype for evaluating task-focused HCP-EHR interaction with real, anonymized patient records.Source: §1 · p. 2
  • The study targets realistic clinical care planning by comparing ClinicalVis with a baseline system that emulates commercially available EHR interfaces.Source: §1 · p. 2
  • The evaluation found generally poor forecasting accuracy, improved HCP experience without overcoming information overload, and more time spent validating care plans with ClinicalVis.Source: §1 · p. 2
  • The paper contributes an open-source system, a proof-of-concept study with practicing HCPs and real patient data, and design insights for task-focused EHR evaluation.Source: §1 · p. 2

2 RELATED WORK

Prior work frames ICU care planning as cognitively demanding because clinicians must integrate irregular, heterogeneous information quickly. Existing evaluations and systems motivate a task-focused study of EHR visualization using realistic clinical work.

Care Planning Complexity in the ICU

  • ICU care planning requires rapid interpretation of irregular, noisy, multimodal data, while fragmented displays and alert fatigue can contribute to cognitive overload and safety threats.Source: Care Planning Complexity in the ICU · p. 2

Supporting HCP’s Information Needs

  • EHRs combine information from many sources and can create mistaken identification, incomplete visibility, and prolonged searching, making customizable displays important across users and scenarios.Source: Supporting HCP’s Information Needs · p. 2

Task Based Evaluation of Care Planning

Visualization-based Explorations of EHR Design

ClinicalVis applies visualization and participatory design to make multimodal EHR information easier to inspect during realistic, time-constrained care-planning tasks. Its interface organizes recent clinical data into linked modules and a shared timeline.

Visualization-based Explorations of EHR Design

  • Visualization research motivates ClinicalVis by showing that visual summaries can highlight patterns, reduce cognitive load, and support decision making, although many systems remain specialized or difficult to reproduce.Source: Visualization-based Explorations of… · p. 3
  • ClinicalVis differs from prior EHR visualizations by evaluating simple, time-constrained displays during simulated clinical tasks with real patient data and no prompts or notifications.Source: Visualization-based Explorations of… · p. 4, §3 · p. 4

ClinicalVis Design Process

  • The design process grounded ClinicalVis in HCP workflows, emergent workplace practices, and multidisciplinary internal validation while targeting common EHR-summary challenges.Source: ClinicalVis Design Process · p. 4

ClinicalVis Interface and Interactions

Baseline GUI Design

The study compares ClinicalVis with a matched baseline while constraining interfaces and data to isolate task-focused HCP-EHR interaction. It uses MIMIC-III ICU records and asks HCPs to forecast near-term ventilation or vasopressor needs.

Baseline GUI Design

  • The baseline GUI matches ClinicalVis in information and interaction capabilities while modeling commercially available EHR systems used by the authors.Source: Baseline GUI Design · p. 4
  • Both interfaces show one patient on a single page using curated, pre-reviewed vitals and real unedited clinical data, while excluding spatial and imaging data but retaining related text.Source: Baseline GUI Design · p. 4

4 RESEARCH SETTING AND METHODS

  • The evaluation asks how HCPs engage with visual EHR representations and whether they affect care-planning accuracy, confidence, and time-to-decision.Source: §4 · p. 4

Data Source

  • MIMIC-III supplies publicly available records from over 58,000 admissions, and this study uses them to evaluate HCP interaction with EHR systems during care planning.Source: Data Source · pp. 4–5

Task Definition

  • The simulated eICU task focuses on forecasting physiological decompensation requiring mechanical ventilation or vasopressors.Source: Task Definition · p. 5
  • The proof-of-concept includes one training record, two think-aloud records, and eight positive and eight control patients selected from ICU records meeting prespecified intervention and documentation criteria.Source: Task Definition · p. 5

Participant Cohort

The study recruited 14 ICU-experienced clinicians and evaluated ClinicalVis and a baseline interface in a structured laboratory study using real, de-identified patient data.

Participant Cohort

  • 14 clinicians with prior ICU experience completed the laboratory evaluation, with specialization diversity controlled during recruitment.Participants averaged 9 hours per week in the ICU and all completed the study.Source: Participant Cohort · p. 5

Study Design

  • The mixed-methods study examined HCP-EHR interaction during task-focused care planning in a remote eICU scenario.The evaluation used ClinicalVis and baseline prototypes with real patient data.Source: Study Design · p. 5, §1 · p. 5
  • Each session introduced participants to both prototypes and the task (5 minutes), followed by think-aloud evaluation (10 minutes), usability and TLX assessment (35 minutes), and comparative survey (10 minutes).Source: Study Design · p. 5

1. Training.

  • Participants received training on both prototypes and the care-planning task before evaluation began.The interfaces were introduced in counterbalanced order using the same patient record.Source: §1 · p. 5, §2 · p. 5

2. Task Introduction.

Participants reviewed ICU records in a time-sensitive eICU scenario, deciding whether to prepare for vasopressors or ventilation while thinking aloud and rating workload.

2. Task Introduction.

  • The study flow included think-aloud evaluation, task-focused usability testing with TLX surveys, and post-completion comparative feedback.Baseline and ClinicalVis were compared through a scripted experimental sequence.Source: Figure 2 · p. 6, §3 · p. 6, §6 · p. 6
  • Participants decided whether each patient might require a vasopressor, ventilator, both, or neither within the next 12 hours, while reporting confidence.The scenario limited participants to 24 hours of available vitals, labs, and notes.Source: §2 · pp. 5–6

3. Think aloud.

  • Participants reviewed eight patient records per interface, with prototype and case ordering counterbalanced using Latin Squares.Each case provided 120 seconds for a decision, with resets permitted in 120-second increments.Source: §4 · p. 6

4. Usability Evaluation and 5. TLX Survey.

  • For each intervention, HCPs submitted a yes/no decision and confidence rating through an evaluation screen at the two-minute mark.Confidence used a four-point Likert scale without a neutral choice.Source: §4 · p. 6
  • After eight assessments, participants rated mental, physical, effort, temporal, performance, and frustration dimensions using 10-point NASA-TLX scales.They also compared the prototypes and indicated which better supported the task.Source: §4 · p. 6, §6 · p. 6

6. Comparative Survey.

Across 224 simulated encounters, ClinicalVis produced only a small, nonsignificant performance advantage, while improving confidence and reported experience relative to the baseline.

5 RESULTS AND DISCUSSION

  • The evaluation comprised 224 simulated EHR-HCP encounters, evenly divided between the two prototypes.Accuracy, confidence, and time-to-task were summarized as quantitative outcomes.Source: §5 · p. 7

EHR-To-Task Performance is Generally Poor

EHR Are Fundamentally Limited For Care Planning.

  • Seven participants said not seeing the patient reduced confidence and led them to reread, overthink, or overinterpret available records.The findings connect care-planning difficulty to the limits of remote EHR information rather than visualization alone.Source: EHR Are Fundamentally Limited For Care… · p. 7

6. Comparative Survey.

  • ClinicalVis was preferred across all TLX dimensions, as shown in Figure 5.Source: TLX analysis · p. 8

EHR Are Fundamentally Limited For Care Planning.

Fundamental Data Concerns that affect HCPs.

HCPs’ decisions were constrained by incomplete and inconsistently documented data, while ClinicalVis changed their experience of reviewing information more than their overall performance.

Fundamental Data Concerns that affect HCPs.

  • Participants looked to recent hours because nursing observations were not entered simultaneously, making temporal interpretation part of care planning.One participant specifically focused on the last four hours because documentation timing varied.Source: Fundamental Data Concerns that affect… · p. 8
  • Insufficient data, inconsistent presentation, and nonstandardized logging reduced confidence during interactions with both prototypes.Participants differed on the preferred aggregation interval for urine output, and some were unfamiliar with note abbreviations.Source: Fundamental Data Concerns that affect… · p. 8

Better Visualization Improved HCP Experience

  • ClinicalVis had little effect on overall performance but was preferred for lower cognitive load, fewer error opportunities, greater content engagement, and information retrieval.The experience advantage was reflected in participant reports and TLX responses.Source: Better Visualization Improved HCP… · p. 8, Figure 5 · p. 8

TLX analysis.

ClinicalVis reduced the perceived effort and negative affect of EHR-based care planning while increasing confidence and changing how HCPs engaged with patient information.

TLX analysis.

  • ClinicalVis integrated into physician workflows, enabling clearer narratives, faster question answering, and lower-effort identification of outliers and relevant information.All 14 participants reported spending less attention on the interface and more on the data; nine described baseline performance as mentally fatiguing.Source: Experiencing EHRs differently · p. 8

Experiencing EHRs differently.

  • HCPs preferred ClinicalVis across four comparative measures: feeling rushed, required work, frustration, and task support.Two participants attributed effort to the patient cases rather than either interface.Source: Altered perception of content · p. 8

Altered perception of content .

  • Baseline use produced anxiety about finding relevant information and missing data, while ClinicalVis made participants feel more confident in their assessments.Participants described the visualization as more reassuring and less controlling than the Baseline.Source: Altered perception of content · pp. 8–9
  • The Baseline scored higher on the first three negative-affect comparisons, whereas ClinicalVis scored highest on the positive-affect comparison.Source: Figure 6 · p. 9
  • Participants described ClinicalVis using data-centered terms such as “spikes” and “trending up,” suggesting faster information consumption than with the Baseline.All participants found notes easier to read, and four incorrectly assumed the notes had been modified for readability.Source: Altered perception of content · p. 9
  • Clinical notes remained central to chart review, but ClinicalVis users interspersed notes with vitals and labs to answer emerging questions.Physicians moved between the timeline and notes modules, using the modular display to contextualize notes and validate intuitions.Source: Clinical Confidence in Visualization · p. 9, Altered perception of content · p. 9

Clinical Confidence in Visualization.

ClinicalVis shifted HCP time from locating information toward validating care plans, although better visualization did not improve accuracy and individual caution shaped planning decisions.

Clinical Confidence in Visualization.

  • ClinicalVis supported faster location of trends and information through a single-screen modular interface that linked notes, vitals, and labs.Participants used notes to answer questions arising from the chart and moved quickly between the patient timeline and notes.Source: Clinical Confidence in Visualization · p. 9

Maximizing Time For Decision Validation, Not Information Finding

  • The Figure 7 comparison concerns per-participant task time across Baseline and ClinicalVis, with more Baseline requests for additional time.Source: Figure 7 · p. 9
  • ClinicalVis users spent available time validating decisions, whereas Baseline users spent it looking for information; average time-to-task therefore inadequately measured performance.Participants reported digesting data more completely in ClinicalVis and using remaining time for validation.Source: Inadequacies of Time to Task metric · pp. 9–10, Maximizing Time For Decision… · p. 9

Inadequacies of Time to Task metric.

  • Active reading and routine reflection did not increase outcome accuracy, while longer task time and lower control-case accuracy suggested more false positives from cautious planning.Participants attributed this pattern to preparing for uncertain prognoses and increased clinical preparedness.Source: Influence of individual assessments of… · p. 10

Influence of individual assessments of caution.

Validation practices that support heuristic methods.

ClinicalVis supported iterative validation of clinical judgments and recall, but the study remains an early task-focused evaluation with limited data scope, tasks, participants, and design process.

Validation practices that support heuristic methods.

  • ClinicalVis users changed decisions while traversing the visualization, recalled patient information better at evaluation, and twelve participants found clinical narratives easier to establish.Source: Validation practices that support… · p. 10

6 LIMITATIONS AND FUTURE WORK

  • The study is a first step toward evaluating visualization systems in clinicians’ task-oriented settings and environmental conditions.Source: §6 · p. 10
  • The study limited displayed data to direct indicators for selected care-planning tasks and calls for broader data sources, time scales, and sampling differences.Source: §6 · p. 10
  • ClinicalVis used simple design principles without an extensive design-focused process, leaving alternative representations of HCP denotations and reading practices for future study.Source: §6 · p. 10
  • The evaluation covered two tasks in one care-planning setting with a limited HCP sample, requiring broader tasks, scenarios, and participant populations for validation.Source: §6 · p. 10

7 CONCLUSION

  • The conclusion characterizes ClinicalVis as improving data experience and confidence without significantly changing performance, while redirecting time toward decision validation.Source: §7 · p. 10
  • ClinicalVis is not custom-fit to one task, can scale to additional variables, and is open-sourced to support broader evaluation of clinician–patient-data interactions.Source: §7 · pp. 10–11

A BASELINE OVERVIEW

The study’s Baseline emulated commercial EHRs, while the task materials represented heterogeneous temporal clinical data and used think-aloud prompts with real patient cases.

A BASELINE OVERVIEW

B DATASET STATISTICS

  • The interfaces contained categorical, numerical, textual, and free-form clinical data with temporal attributes attached to observations or note sets.Source: B DATASET STATISTICS · p. 14

C STUDY PROMPTS

  • Study prompts asked participants to verbalize what they saw, what they sought, and why they expected a particular action or outcome.Source: C STUDY PROMPTS · p. 14
  • The baseline interface displayed fake patient data in the figure, although participants saw real patient data during the study.Source: Figure 8 · p. 15
  • The study used real patient cases selected for care-planning tasks, with reported patient identifiers, demographics, ICU types, and vasopressor or ventilation outcomes.Source: Table 4 · p. 15, Table 5 · p. 15

Related papers

Loading 1810.05798v1…