PaperlayerView exact revision on arXiv

Source-linked AI summary

The Role of Mixed and Augmented Reality in Medical Visualization: Literature Review and A Context-Aware Taxonomy

Xinrui Zou, Mingxu Liu, Hongchao Shu, Ruixing Liang, Mathias Unberath, Alejandro Martin-Gomez

arXiv:2608.27644v1cs.GR

TL;DR

Traditional two-dimensional displays leave clinicians to mentally reconstruct spatial relationships in three-dimensional medical data. This paper systematically reviews medical AR/MR visualization and organizes systems in a six-dimension taxonomy; its analysis identifies persistent challenges in modality integration, perceptual design, and soft-tissue adaptability.

  • Problem

    Traditional two-dimensional displays of three-dimensional medical data require clinicians to mentally reconstruct spatial relationships.

    Source: §1 · p. 2

  • Method

    The authors conduct a systematic literature review and classify medical AR/MR visualization systems across six dimensions: application, image modality, dimensionality, display technology, visualization anchoring, and perceptual support.

    Source: §1 · p. 2, §12 · p. 9

  • Results

    The analysis identifies persistent challenges in modality integration, perceptual design, and adaptability to soft-tissue dynamics.

    Source: §12 · p. 9

  • Takeaways & Limitations

    The taxonomy is intended to support transparent reporting, cross-context comparison, and identification of underexplored design combinations.

    Source: §12 · p. 9

    The taxonomy is descriptive rather than evaluative, so observed patterns between visualization strategies and clinical contexts do not establish which choices are more effective.

    Source: §11.3 · p. 9

Abstract

from arXiv · show

The discovery and evolution of medical imaging technologies have enabled non-invasive visualization of internal anatomy that has be- come essential for supporting diagnosis, monitoring, and treatment. However, because medical imaging relies on complex physical processes and contrast mechanisms for image formation, imaging alone is not sufficient to enable humans to leverage the resulting in- formation fully. In addition, traditional methods to visualize the resulting information use two-dimensional displays to present three-dimensional anatomical structures. The introduction of Augmented and Mixed Reality (AR/MR) technologies offers an opportunity to provide valuable paradigms for medical imaging visualization, allowing users to observe, explore, and interact with anatomical infor- mation in more spatially intuitive ways. However, naive implementation without careful design considerations can lead to perceptual inconsistencies, potentially compromising utility and effectiveness. In this paper, we present a structured taxonomy of medical AR/MR visualization strategies aimed at providing clearer insight into how visualization design varies across clinical use cases. The taxonomy organizes techniques based on four core design components: image modality, data dimensionality, display technology, and clinical application. In addition, we introduce two critical dimensions that are often overlooked in the literature: visualization anchoring (the spatial relationship between virtual content and the physical world), and perceptual awareness (the use of visual cues to support spatial interpretation). Together, these components form a comprehensive taxonomy, offering a detailed framework for selecting appropriate visualization techniques in medical applications.

1 INTRODUCTION

The introduction argues that medical images need effective visualization and that 2D displays make 3D anatomy difficult to interpret. It presents a context-aware taxonomy for medical AR/MR visualization that connects design choices to clinical tasks and includes anchoring and perceptual awareness.

1 INTRODUCTION

  • Medical images require visualization techniques that transform data into meaningful representations because imaging alone does not let people fully use the information it contains.Such representations support anatomical interpretation, diagnosis, surgical planning, and image-guided procedures.Source: §1 · p. 1
  • Traditional 2D displays require clinicians to mentally reconstruct spatial relationships in 3D medical data, motivating displays that present anatomy more accurately.Even digital systems offered limited dynamic interaction with 3D data through 2D media.Source: §1 · p. 2
  • AR/MR place virtual content in the physical environment for real-time interaction with patient-specific anatomy, while VR’s separation from the real world limits anatomy-aligned tasks.The introduction describes MR as a more robust form of AR with advanced environmental understanding that lets virtual content interact with users and surroundings.Source: §1 · p. 2
  • AR/MR design must balance spatial accuracy, visual clarity, and usability because suitable visualization strategies depend on clinical workflows, user needs, and task demands.The introduction notes that the relationship between clinical context and visualization selection has not been systematically reviewed.Source: §1 · p. 2
  • The taxonomy organizes medical AR/MR visualization by clinical context, image modality, data dimensionality, display technology, and clinical use case, adding anchoring and perceptual awareness.Anchoring describes the spatial relationship between virtual content and the real world; perceptual awareness concerns visual cues that support spatial interpretation.Source: §1 · p. 2

2 RELATED WORK

Existing AR/MR reviews examine clinical applications or technical design, but provide limited guidance on how visualization strategies should vary with clinical objectives. This paper addresses that gap with a taxonomy connecting design dimensions to medical use cases and emphasizing anchoring and perceptual awareness.

2 RELATED WORK

  • Few studies systematically examine how clinical objectives shape visualization design, especially spatial integration and perceptual support.Prior reviews tend to classify AR/MR by usage, such as clinical tasks, or design, such as general pipelines.Source: §2 · p. 2
  • Reviews of AR/MR in orthopedics, plastic surgery, education, and surgical guidance prioritize procedural outcomes over visualization design.They offer insights into system performance, clinical feasibility, and workflow integration, but leave visualization adaptation to clinical scenarios poorly understood.Source: §2 · p. 2
  • Existing frameworks characterize data-to-display pipelines, visual properties, or display hardware and viewpoints, but do not connect these layers to clinical context.They omit explicit links to medical tasks, image modality, anatomical constraints, anchoring strategy, and perceptual support, limiting guidance on context-specific choices.Source: §2 · p. 2
  • The proposed taxonomy links image modality, dimensionality, and display technology to medical use cases while emphasizing visualization anchoring and perceptual awareness.These dimensions address content placement and spatial interpretation in clinical AR/MR systems.Source: §2 · p. 2

3 DATA COLLECTION

The review systematically searched major medical and technical databases for peer-reviewed studies of AR/MR medical-image visualization in clinical applications, then screened and classified eligible studies using a shared taxonomy template.

3 DATA COLLECTION

  • PubMed, IEEE Xplore, and Scopus were searched for studies describing the representation, rendering, or integration of medical imaging data with AR/MR in clinical applications, following applicable PRISMA 2020 items.The database-specific search strings and filters are reported in the supplementary material.Source: §3 · pp. 2–3
  • Two reviewers independently screened titles and abstracts, then assessed full texts; exclusions covered out-of-scope or insufficiently detailed studies, leaving 313 studies in the final review.The search covered studies published before February 5, 2026, and retained peer-reviewed original articles while excluding reviews and meta-analyses where database filters allowed.Source: §3 · p. 3
  • Four authors extracted data at full-text stage, with each study reviewed by two authors and classification disagreements resolved through discussion.A shared template defined taxonomy dimensions and decision rules; categories were assigned from each paper’s system description, figures, and implementation details.Source: §3 · p. 3

4 TAXONOMY

The taxonomy provides a six-dimension framework for analyzing and selecting medical AR/MR visualization techniques, combining core design choices with considerations for integration into clinical environments.

4 TAXONOMY

  • The four core dimensions specify the clinical context, imaging data, spatial structure, and hardware for presenting augmented content.The review includes optical imaging for anatomical visualization in clinical or educational use, and dimensionality can include time-resolved 4D content.Source: §4 · p. 3
  • The taxonomy organizes medical AR/MR visualization by application, image modality, dimensionality, display technology, visualization anchoring, and perceptual awareness.Figure 3 presents these six design considerations together.Source: Figure 3 · p. 3, §4 · p. 3
  • Visualization anchoring describes whether content appears in-situ, inter-situ, or off-situ, distinguishing its placement in the user’s view from geometric registration.Registration maps image-derived content geometrically to the patient or scene; anchoring describes the resulting placement of visual content.Source: §4 · p. 3
  • Perceptual awareness incorporates visual-perception principles to support depth perception, spatial alignment, scene understanding, and visual consistency.Such techniques can contribute to greater task accuracy and system acceptance in complex procedures.Source: §4 · p. 3
  • The Application dimension is presented last in Section 10 to synthesize findings from a clinical-workflow perspective and provide a context-aware overview.Application refers to the clinical task and medical specialty, whose differences affect visualization requirements.Source: §4 · p. 3

5 IMAGE MODALITY

Medical AR/MR modality choice depends on clinical context: CT or X-ray is commonly used for rigid bone, MRI for soft tissue, ultrasound for moving or deforming anatomy, and multimodal imaging can combine rigid and deformable structures [12].

Modality selection

  • Clinical context determines which imaging modality AR/MR presents, with multimodal combinations supporting visualization of both rigid and deformable structures [12].CT or X-ray is commonly used for rigid bony anatomy, MRI for soft tissue, and ultrasound for moving or deforming anatomy.Source: §5 · pp. 3–4

Radiography (X-rays)

  • AR/MR places 2D X-ray projections in 3D context to reduce mental effort and improve scene understanding and coordination, with fewer repeat acquisitions reported in some settings.Because the image remains projection-based, depth inference and occlusion reasoning depend heavily on visualization and interaction design.Source: §5 · p. 4

Computed Tomography (CT)

  • CT is the most prevalent medical AR/MR modality, supporting surface and volumetric models and usually using preoperative scans for planning, guidance, and education.Point-of-care or intraoperative updates are possible, but intraoperative CT is constrained by cost, radiation exposure, and operating-room demands.Source: §5 · p. 4

Magnetic Resonance Imaging (MRI)

  • MRI is used when soft-tissue visibility is crucial, commonly in neurosurgery, but intensity variability and lower resolution and signal-to-noise ratio often require preprocessing or segmentation.Intraoperative MRI can update anatomy and reduce mismatch from brain shift or resection, but is less frequent because of cost, acquisition time, and infrastructure requirements.Source: §5 · p. 4

Ultrasound

  • Ultrasound supports real-time guidance for moving or deforming anatomy; AR/MR commonly overlays 2D images, while some systems reconstruct 3D volumes.Image quality can limit interpretability, motivating shading enhancements; despite use in fetal and cardiac imaging, 4D ultrasound remains rare in AR/MR, likely due to computational and visualization demands.Source: §5 · p. 4

Optical Imaging

  • Optical imaging most often supplies endoscopic video as a contextual background for overlays rather than a standalone taxonomy modality, though some systems reconstruct and augment the operative scene.Less common directions include hyperspectral imaging for tissue classification and optical coherence tomography for microstructural cross-sectional visualization.Source: §5 · p. 4

6 DIMENSIONALITY

Dimensionality shapes what medical AR/MR systems can render and register, influencing representation and processing choices. The section focuses on prevailing 2D and 3D approaches, while noting barriers to broader 4D adoption.

Dimensionality

  • 4D imaging can represent time-varying anatomy, including respiratory motion or cardiac cycles, but adoption is limited by acquisition, processing, and workflow burdens.Accordingly, the section concentrates on prevailing 2D and 3D representations and their common processing choices.Source: §6 · p. 4

Three-dimensional

  • 3D visualization is predominant in medical AR/MR because it supports spatial interaction and anatomical exploration for planning, training, and education.Volume rendering maps voxel intensities to color and transparency but is computationally expensive; surface modeling creates lightweight interactive meshes and is more common, but depends on segmentation quality, with errors risking misleading visualization. Applications include surgical planning, procedural training, and medical education.Source: §6 · p. 4

Two-dimensional

  • 2D images serve as anatomical references or virtual panels, offering practical intraoperative guidance and accessible context despite lacking depth.References such as X-rays, 2D ultrasound, and CT/MRI slices may be overlaid on the patient or positioned in space; panels can combine images with vital signs, procedural steps, or annotations and support ergonomics and cognitive flow through flexible placement.Source: §6 · p. 4

7 DISPLAY TECHNOLOGY

Display choice is shaped by clinical workflow constraints and determines how augmented information is placed, how depth and occlusion are handled, and how much perceptual support is needed. The taxonomy groups devices as head-mounted, hand-held, or spatial displays.

Display technology

  • Display selection must account for sterility, hands-free use, line-of-sight, team-shared viewing, and latency, while shaping placement, occlusion, depth consistency, and perceptual support.The taxonomy groups devices by usage context into head-mounted, hand-held, and spatial displays.Source: §7 · p. 4
  • Head-mounted displays support hands-free, user-centered viewing, but optical and video see-through designs have distinct benefits and risks.OST-HMDs preserve direct sight of the field but have restricted field of view, reduced contrast, calibration sensitivity, discomfort, and unreliable real-object occlusion. VST-HMDs enable controlled occlusion and more consistent depth composition, but latency or capture, tracking, and rendering failures can disrupt hand-eye coordination [10].Source: §7 · p. 4
  • Spatial displays support team viewing and fit established workflows, but can introduce spatial mismatch, mobility limits, or viewpoint-sensitive distortion.Monitors integrate into endoscopy, fluoroscopy, and robotic surgery, though patient-display mismatch can increase cognitive load; stereo options face eyewear, viewing-zone, and resolution constraints. Microscopes provide high optical fidelity and view-aligned overlays, while projection systems can cause parallax or distortion without robust calibration, and beam-splitter viewing range can degrade with angle changes.Source: §7 · p. 5
  • Hand-held devices prioritize portability and low deployment cost for point-of-care viewing and education, but require manual interaction that can conflict with sterile, hands-free workflows.Source: §7 · p. 5

8 VISUALIZATION ANCHORING

Visualization anchoring classifies how medical AR/MR content relates spatially to the patient and operative scene: off-situ, inter-situ, and in-situ represent increasing spatial coupling. This choice shapes attention switching, hand–eye coordination, registration needs, and reliance on perceptual cues.

8.1 Off-situ

  • Off-situ content is spatially separate from the patient, requiring attention shifts and mental mapping; it appears in about 78% of reviewed medical-education literature and supports safe, repeatable exploration.Examples include external monitors, floating 3D models, and fixed-screen-space virtual panels; the approach avoids registration but can increase cognitive load.Source: §8.1 · p. 5

8.2 Inter-situ

  • Inter-situ content remains outside the immediate physical space but references real-world objects, reducing obstruction while preserving spatial alignment and enabling collaboration and flexible modification.Examples include mirror-based approaches and duplicated AR; separately positioned, rotationally aligned models increased user preference.Source: §8.2 · p. 5

8.3 In-situ

  • In-situ visualization overlays 2D or 3D content on anatomy, supporting planning and guidance; more than 50% of surveyed systems used it, especially in clinical applications.It reduces hand–eye coordination demands, but depends on accurate registration; misalignment can distort spatial interpretation, while poorly handled occlusion and depth cues can make content appear detached.Source: §8.3 · p. 5

9 PERCEPTUAL AWARENESS

Effective AR/MR medical visualization requires visual cues that help users interpret virtual–physical spatial relationships, beyond precise registration or stereoscopic depth alone. The section groups these cues as pictorial, kinetic, and illustrative strategies.

9 PERCEPTUAL AWARENESS

  • Even precise registration can mislead users when occlusion or depth cues are weak, so AR/MR visualization must integrate virtual elements with the physical scene.Such perceptual errors can compromise clinical tasks, including interpretation of anatomy and tool placement.Source: §9 · p. 5

9.1 Pictorial Cues

  • Monocular pictorial cues help users infer depth and 3D relationships, and unlike stereopsis, work across AR/MR display technologies.Source: §9.1 · p. 5

9.1.1 Occlusion

  • Occlusion is the strongest pictorial depth cue, but rendering internal anatomy atop skin can make structures appear outside the body.Virtual cutaways and random-dot masks reveal hidden anatomy but may obscure surfaces or structures; contours, semi-transparent overlays, and foreground segmentation preserve depth and manage occlusion.Source: §9.1.1 · pp. 5–6

9.1.2 Texture

  • Texture gradients can encode depth in medical visualization, but texture alone has limited or no significant depth-perception benefit; chromadepth can complement it.Source: §9.1.2 · p. 6

9.1.3 Illumination

  • Shadows on reference planes convey virtual objects’ position relative to the physical environment, while augmented mirrors offer alternative anatomical-overlay views in laparoscopic surgery.The taxonomy treats shadows and reflections as pictorial cues that support depth, shape, and spatial understanding.Source: §9.1.3 · p. 6

9.2 Kinetic Cues

  • Kinetic cues use observer, scene, or object motion to support depth judgments; viewpoint-driven focus modulation and animated depth representations provide dynamic alternatives to static cues [70].Source: §9.2 · p. 6

9.3 Illustrative Cues

  • Illustrative cues deliberately encode spatial relationships: chromadepth maps nearer and farther anatomy to colors, while numeric measurements and markers communicate distance and scale.Source: §9.3 · p. 6

10 APPLICATION

The application taxonomy organizes studies by clinical workflow task and, for intraoperative systems, by specialty. It finds that intraoperative use dominates, while visualization choices and perceptual challenges vary with the task and surgical context.

10 APPLICATION

  • 66.5% of the 313 included studies address intraoperative applications, followed by preoperative planning (15.7%), medical and patient education (6.4%), and simulation-based training (4.2%).The review reports within-category proportions because category sizes are imbalanced; sparse or non-workflow studies are excluded from detailed comparison.Source: §10 · p. 6

10.1 Medical and Patient Education

  • Educational systems commonly use pre-generated CT- or MRI-derived 3D models for interactive exploration, prioritizing conceptual understanding over precise alignment.AR/MR can ground learning in physical bodies, manikins, or clinical tools; perceptual support is rarely explicit, with understanding more often supported by simplified presentation and interaction.Source: §10.1 · p. 7

10.2 Training and Simulation

  • AR/MR training is relatively scarce but supports procedural rehearsal with physical instruments, models, or task environments, including tumor-resection rehearsal and AI-based action tracking with performance feedback.Training emphasizes tool coordination, line-of-sight management, and interpreting overlays under imperfect alignment; inter-situ setups can support progressive, error-tolerant practice before in-situ workflows.Source: §10.2 · p. 7

10.3 Preoperative Planning

  • Preoperative planning commonly turns CT or MRI into rotatable, measurable, annotatable 3D models, with more than half of systems using off-situ integration that avoids patient registration.Clear, controllable views support trajectory planning and collaborative surgical discussion, although explicit perceptual-awareness strategies remain limited despite the need to understand spatial relationships.Source: §10.3 · pp. 7–8

10.4 Intraoperative Use

  • Most intraoperative systems use in-situ CT imaging and 3D surface-model overlays on monitors or OST-HMDs, though few explicitly address depth interpretation.Depth ambiguity, occlusion conflicts, and visual clutter may reduce guidance effectiveness; goals include information display, surgical guidance, and patient positioning.Source: §10.4 · p. 8
  • Neurosurgical systems more often combine MRI with CT and use in-situ 3D overlays, while orthopedic systems primarily use CT and X-rays, sometimes displaying 2D fluoroscopy or X-rays in the augmented view.Neurosurgical overlays must avoid clutter, misleading occlusion, and depth-ordering ambiguity. Orthopedic visualization is geometry-driven around rigid bone; spatially relating 2D images to the surgical field may reduce mental reconstruction from fluoroscopic views.Source: §10.4 · p. 8
  • Head & Neck has the highest proportion of in-situ use, whereas thoraco-abdominal systems use in-situ and off-situ equally often and more frequently use ultrasound and volume rendering.Rigid bony anatomy and accessible facial or dental references facilitate Head & Neck registration, but close-range overlays remain vulnerable to clutter, occlusion, and depth-ordering problems. Deformable, motion-prone thoraco-abdominal anatomy may favor volumetric context and cues more tolerant of registration uncertainty.Source: §10.4 · p. 8

11 DISCUSSION AND FUTURE DIRECTIONS

The discussion argues that medical AR/MR visualization design is underprioritized, especially perceptual support, and identifies deformable anatomy as a major challenge. It offers context-aware heuristics as reflective prompts, not evaluative benchmarks or universal standards.

11.1 Visualization Underprioritized in System Design

  • Visualization remains underemphasized: generic surface models and standard rendering are common, while perceptual cues are limited despite their role in spatial interpretation.Cue selection affects depth judgment and response time in neurosurgical visualization, while display and interaction design influence mental effort in planning; the taxonomy supports reflection on design choices but is not an evaluative framework.Source: §11.1 · pp. 8–9
  • OST-HMD use rose from 31.0% to 59.6%, while studies with identifiable perceptual cues fell from 25.9% to 8.8%.The authors caution that reporting bias, terminology differences, shifting priorities, or insufficient methodological and visual detail may affect this temporal pattern; OST-HMDs also struggle with occlusion and depth consistency.Source: §11.1 · p. 9

11.2 Deformable Tissue and Dynamic Anatomy

  • Deformable tissues can shift, stretch, or compress during procedures, making preoperative anatomy inaccurate for augmentation, especially in variable organs such as the liver and lungs.NeRF and 3D Gaussian Splatting offer possible reconstruction approaches, with real-time rendering, deformable modeling, medical reconstruction, and AR/MR integration explored; clinical use faces dense-data, computational, calibration, and robustness barriers.Source: §11.2 · p. 9

11.3 Context-Aware Design Heuristics

  • The descriptive taxonomy reveals recurring links between visualization strategies and clinical contexts, but does not establish which choices are more effective.The proposed heuristics are design prompts derived from observed practices, not definitive benchmarks or universal standards; the taxonomy provides a foundation for reflecting on modality, task, and user needs.Source: §11.3 · p. 9, §11.1 · p. 9
  • Preserve modality-specific strengths by selecting representations that retain defining image characteristics, rather than applying a uniform rendering strategy.Surface models can delineate high-contrast CT boundaries, complemented by volume rendering for subcortical or occluded anatomy; rigid surfaces may oversimplify MRI’s nuanced soft-tissue contrast.Source: §11.3 · p. 9
  • Match display and interaction choices to clinical workflow: hands-free, spatially aligned views may suit continuous alignment, while handheld or monitor views may support planning and shared review.Task timing, sterility, and collaboration shape these choices.Source: §11.3 · p. 9
  • When depth ambiguity or registration uncertainty threatens safety or performance, use occlusion management, dynamic visualization, or illustrative cues to improve spatial interpretation.The discussion identifies accurate depth estimation as a major challenge in AR-based medical visualization.Source: §11.3 · p. 9

12 CONCLUSION

The paper presents a six-dimension taxonomy of medical AR/MR visualization based on a systematic literature review. The review identifies persistent design challenges and proposes ways the taxonomy can guide reporting, comparison, and exploration of underused design combinations.

12 CONCLUSION

  • The taxonomy classifies medical AR/MR visualization by application, image modality, dimensionality, display technology, visualization anchoring, and perceptual support.It is based on a systematic literature review examining how design choices relate to clinical tasks and identifying trends and gaps.Source: §12 · p. 9
  • The review identifies persistent challenges in modality integration, perceptual design, and adapting AR/MR visualization to soft-tissue dynamics.Addressing these gaps calls for visualization-centered development and closer collaboration between technical designers and clinical users.Source: §12 · p. 9
  • The taxonomy is intended to support transparent reporting, cross-context comparison, and identification of underexplored design combinations.One example is using volume rendering more often in oncology workflows where lesion boundaries are unclear and surface models can be misleading.Source: §12 · p. 9

Related papers

Loading 2608.27644v1…