Code Black: Desktop-Mediated Co-Design of AR-HMD Microinteractions for Emergency Department Teamwork

Jonathan Segal; Jalynn Nicoly; Francisco Ortega; Angelique Taylor

Source-linked AI summary

Code Black: Desktop-Mediated Co-Design of AR-HMD Microinteractions for Emergency Department Teamwork

Jonathan Segal, Jalynn Nicoly, Francisco Ortega, Angelique Taylor

arXiv:2609.03295v1Sep 3, 2026cs.HC

TL;DR

ED teams need interfaces that support shifting roles, timed interventions, and medication decisions without disrupting safety-critical coordination. The study used speculative desktop-mediated co-design with HCWs to refine AR-HMD concepts and derive microinteraction specifications. It produced SCF-HMD and a visual design catalog, while identifying tensions that require testing beyond speculative design.

Problem
Limited evidence guides co-design of AR-HMD interfaces for dynamic, safety-critical ED teams while accounting for work-as-done constraints and situated interaction behavior.
Method
The study used an editable desktop-mediated Unity 3D probe with microinteraction prompts, involving 12 HCWs in co-design and 31 HCWs in follow-up feedback.
Results
The study elicited formative specifications for role visibility, task-linked timing, and verification-oriented dosage assistance, while revealing tensions involving clutter, awareness, communication, privacy, and reliability.
Takeaways & Limitations
SCF-HMD translates expert critique of work-as-imagined AR-HMD concepts into situated goals for future spatial interfaces before functional implementation.
Takeaways & Limitations
The study used a desktop-mediated probe rather than a functional AR-HMD system and individual HCWs rather than live clinical teams.

Abstract

from arXiv · show

Emergency Department (ED) teams coordinate shifting roles, medication decisions, and time-critical interventions under uncertainty. Augmented reality head-mounted displays (AR-HMDs) have shown potential to spatially anchor information during care, creating opportunities to examine how spatial interfaces might support teamwork. We conducted a speculative co-design study with 12 healthcare workers (HCWs) using an editable, desktop-mediated Unity-based 3D design probe to visualize and refine work-as-imagined AR-HMD interfaces for role-based notifications, task-specific timers, and dosage verification. Guided by microinteraction rules, participants identified future spatial user interfaces (SUI) requirements such as how they appear, update, or are dismissed in relation to clinical practice, safety concerns, and existing tools. Five returning participants and 26 additional HCWs subsequently provided follow-up feedback on derived visual interface alternatives. Findings show that desktop-mediated spatial co-design elicited formative specifications for role visibility, task-linked timing, and verification-oriented dosage assistance, while revealing tensions involving clutter, shared awareness, communication, privacy, and reliability. Rather than evaluating a functional AR-HMD system or team-based clinical performance, this study contributes the Speculative Co-Design Framework for AR-HMD Teamwork (SCF-HMD) and a visual design catalog for translating expert critique of work-as-imagined (WAI) concepts into situated goals for future AR-HMD systems.

1 Introduction

ED teamwork requires shared awareness amid shifting responsibilities, time pressure, and safety-critical decisions, yet AR-HMD team interfaces remain underexplored. This study uses speculative co-design to translate clinician critique into situated design specifications and future development goals.

ED teams struggle to maintain situational awareness because communication, coordination, and information access can break down under high workloads.
Limited research examines AR-HMD interface design for dynamically assembled ED teams whose information needs change with roles, responsibilities, and interventions.
Prior work offers limited guidance for eliciting spatial notifications, collaborative attention, and interaction modalities during co-design.
The study conducted speculative co-design with 12 HCWs using an editable desktop-mediated Unity 3D probe, followed by feedback from five returning and 26 additional HCWs.
The contributions are a formative desktop-mediated co-design method, SCF-HMD for bridging WAI and WAD, and a visual catalog of speculative AR-HMD interface concepts.

2 Background

ED coordination depends on timely shared information without added distraction or communication burden. Participatory, speculative, and microinteraction-based design provide ways to examine how AR-HMD interfaces might support team practice before deployment.

Teamwork in Healthcare: ED teams coordinate shifting responsibilities and next actions through shared understanding and closed-loop communication during time-sensitive care.
Teamwork in Healthcare: Untimely or inaccessible information can disrupt communication, delay action, and compromise patient safety.
AR-HMDs for Teamwork: AR-HMD research has explored presenting clinical information and collaborative cues, but offers less guidance for coordinating several clinicians around changing information.
Speculative and Participatory Design: Participatory design positions HCWs as domain experts, while co-design involves stakeholders in generating, critiquing, and refining technologies or design futures.
Microinteractions: Microinteractions comprise triggers, rules, feedback, loops, and modes that help specify how AR-HMD cues behave across situated clinical situations.

3 Methods

The study used two stages to examine how HCWs could critique and refine work-as-imagined AR-HMD interfaces for ED teamwork before functional development. Stage 1 involved remote desktop-mediated co-design, followed by a synthesized visual catalog and Stage 2 feedback.

Stage 1 involved 12 HCWs remotely co-designing role-awareness, task-timer, and dosage-verification interfaces in an editable Unity-based 3D probe.
Participants compared spatial interface behaviors with existing practices, tools, and safety concerns before the concepts were synthesized into a visual catalog.
Stage 2 included follow-up interviews with five returning participants and a survey completed by 26 additional HCWs.

3.1 Stage 1: Desktop-Mediated 3D Co-Design of Future AR-HMD Interfaces

Stage 1 used microinteraction prompts and an editable desktop-mediated Unity scene to ground speculative AR-HMD concepts in ED coordination challenges. HCWs refined interfaces for role awareness, timed interventions, and dosage verification without wearing headsets or performing simulated treatment.

Design Goals: The co-design sessions examined responsibility for evolving tasks, awareness of recurring interventions, and medication-dosage verification during rapid care.
Design Goals: The three design goals addressed role-based notifications, task-specific timers, and dosage-verification information as starting points rather than fixed solutions.
Participants: The 12 Stage 1 participants included eight attending physicians, one resident physician, one nurse practitioner, and two registered nurses.
Design Probe: Participants viewed an ICU-style Unity scene through desktop screen-sharing and did not wear an AR-HMD or complete a simulated treatment task.
Co-Design Procedure: Researchers introduced microinteraction concepts, then repositioned, resized, added, or removed interface elements in response to participant feedback.
Data Collection and Analysis: Analysis coded triggers, rules, feedback, loops, and modes across role notifications, task timers, and dosage-verification interfaces.

3.2 Stage 2: Follow-Up Feedback on Visual Interface Alternatives

Stage 2 gathered formative feedback on visual alternatives for role-based notifications, task-specific timers, dosage verification, and related ED coordination concerns. Interviews and surveys used complementary analyses to examine preferences, explanations, and context-dependent requirements.

Participants: 31 HCWs provided Stage 2 follow-up feedback through interviews with five returning participants and a survey of 26 additional HCWs.Participants included attending physicians, registered nurses, nurse practitioners, fellows, a resident physician, a physician assistant, and a child life specialist.
Data Collection and Analysis: Some ranking items were omitted, so Stage 2 responses were analyzed using item-specific counts rather than a uniform sample size.The table caption states that response counts vary because some participants omitted individual items.
Data Collection: Participants reviewed or ranked alternatives for role-based notifications, task-specific timers, dosage verification, and an additional intervention-related interface.Returning participants discussed a composite emergency-care scenario, while survey participants ranked corresponding alternatives from most to least preferred.
Data Collection and Analysis: Interview and survey data were analyzed separately, then compared to identify where interview explanations supported, conflicted with, or qualified survey preferences.Survey responses were summarized with first-choice frequencies, percentages, mean rank, and standard deviation; interview data were examined for preferences, concerns, and workflow fit.
Visual Interface Alternatives: Role-based notification alternatives varied label content, clinician-linked placement, and role-based color coding, alongside minimized-notification options.Figure 4 marks these variations with yellow circles and minimized notifications with red squares.

4 Findings

The findings refine three AR-HMD interface areas for ED teamwork: role-based notifications, task-specific timers, and dosage verification. Participants specified visibility, timing, feedback, and verification behaviors while identifying tensions involving clutter, communication, and clinical context.

Role-Based Notifications: Role-based notifications supported awareness of team responsibilities by showing clinician identity, role, specialty, title, or current task.Participants viewed them as potential badge replacements for unfamiliar team members, while retaining verbal role assignment.
Role-Based Notifications: Five of 12 participants preferred persistent notifications, whereas seven favored visibility during team assembly followed by fading until roles or requests changed.Peripheral lists also tracked team members who had been called, were expected, or had temporarily left.
Role-Based Notifications: Name ranked first for 22/26 respondents (84.6%; 𝑀= 1.42, 𝑆𝐷= 1.06), while role and title each ranked first for 2/26 (7.7%).The findings support concise, configurable notifications that adapt visibility to team familiarity, task assignments, and clutter.
Task-Specific Timers: Task-specific timers were designed for CPR, medication administration, patient arrival, and recurring interventions, with six participants preferring edge-of-view or anchored placement over patient overlays.Participants linked timer triggers to events such as code initiation, pulse checks, compressor switches, repeat doses, and defibrillation.
Task-Specific Timers: Participants specified stateful timer feedback and recurring loops, including color changes before completion, flashing at meaningful intervals, and resets tied to elapsed time or clinical action.These designs framed timers as glanceable cues intended to signal urgency without unnecessarily interrupting care.
Task-Specific Timers: Flashing colors ranked first for active timer feedback for 9/19 respondents (47.4%; 𝑀= 1.58, 𝑆𝐷= 0.61), while auditory feedback ranked first for completed intervals for 10/19 (52.6%; 𝑀= 1.68, 𝑆𝐷= 0.82).Interview participants cautioned that auditory feedback could add noise and interfere with team communication.
Dosage Verification: Dosage verification shifted from displaying a calculated dose toward clinician-controlled checking using medication history, patient weight, calculation visibility, and clinical indication.Participants treated the equation and contextual information as aids for verifying outputs rather than replacements for references or double-checking.
Dosage Verification: Participants favored dosage information that connected medication decisions to administration history while remaining retrievable without obstructing care or competing with speech.They proposed triggered medication notifications near the administering clinician that could transition into medication history.

5 Discussion

The discussion positions SCF-HMD as an early-stage method for translating expert critique of speculative AR-HMD concepts into testable, situated development goals. It also emphasizes that desktop-mediated, individual critique cannot establish immersion, usability, safety, effectiveness, or team performance.

SCF-HMD: SCF-HMD bridges work-as-imagined concepts and work-as-done accounts through editable probes and microinteraction prompts.The framework translates critique into spatial and temporal behaviors that developers can later prototype and test.
SCF-HMD: The framework frames team challenges, bounds interventions against existing tools, situates concepts around people, tasks, and equipment, and translates critique into evaluation goals.Its three-phase process includes problem framing, co-design iteration, and refinement toward later evaluation.
Desktop-Mediated Co-Design: Desktop-mediated co-design enabled remote spatial reasoning about placement and behavior without reproducing headset immersion, embodied movement, depth perception, field-of-view limits, or live-care demands.The research team manually manipulated the shared Unity environment in response to verbal feedback.
Team-Level Evaluation: Because participants worked individually, the study could not capture synchronous communication, shared awareness, or social dynamics among multiple HCWs.Live team settings may introduce clutter, blocked eye contact, interruptions to closed-loop communication, or friction between headset wearers and non-wearers.
Institutional Infrastructure: Future prototypes must test institutional constraints including data availability, latency, reliability, authentication, permissions, provenance, freshness, uncertainty, and clinician control.Delayed or incomplete backend data could be worse than no display if clinicians treat the interface as authoritative.
Limitations: The findings are formative refinements rather than consensus on final designs or evidence of usability, partly because the probe lacked functional AR-HMD evaluation and the sample was physician-weighted.A more interprofessionally balanced cohort might produce different requirements for notifications, medication support, documentation, equipment coordination, and interruption tolerance.

6 Conclusion

The study used speculative co-design to translate AR-HMD concepts for safety-critical ED teamwork into practice- and safety-grounded interface specifications. It proposes SCF-HMD as a reusable process for refining spatial interface behavior before implementation, rather than demonstrating deployed-system usability or effectiveness.

Conclusion: SCF-HMD supports developing AR-HMD systems that complement existing high-stakes teamwork rather than assuming augmentation is inherently beneficial.The framework elicits and refines work-as-imagined spatial interface behavior before functional implementation.