PaperlayerView exact revision on arXiv

Source-linked AI summary

Utility of Optical See-Through Head Mounted Displays in Augmented Reality-Assisted Surgery: A systematic review

Manuel Birlo, P. J. "Eddie'' Edwards, Matthew Clarkson, Danail Stoyanov

arXiv:2202.04141v1cs.HC

TL;DR

Despite decades of research, optical see-through head-mounted displays have not translated into routine surgical practice. This systematic review of 2013–2020 studies finds rising publications and application diversity, but few clinical uses and rare demonstrations of utility.

  • Problem

    Despite decades of proposed AR surgical guidance, OST-HMD systems have not translated into routine clinical practice, leaving their applicability and adoption barriers an active review question.

    Source: §1 · p. 1

  • Method

    The authors conducted a PRISMA-guided review of 2013–2020 literature, categorizing applications, registration and validation, visual content, and human factors.

    Source: §1 · p. 1, §3.1 · p. 3

  • Results

    Publications and application diversity have risen, but few OST-HMD systems have been used clinically and utility demonstrations remain rare; phantom experiments dominate, while human factors remain influential.

    Source: §10 · p. 22, §10.3 · p. 24

  • Takeaways & Limitations

    The review suggests identifying procedure-specific moments tied to patient outcomes and tailoring visualizations, with laboratory performance studies helping clarify AR’s role.

    Source: §10.3 · p. 25

    The reviewed systems have not reached the large-scale clinical trials needed to demonstrate patient benefit.

    Source: §2.3 · p. 3

Abstract

from arXiv · show

This article presents a systematic review of optical see-through head mounted display (OST-HMD) usage in augmented reality (AR) surgery applications from 2013 to 2020. Articles were categorised by: OST-HMD device, surgical speciality, surgical application context, visualisation content, experimental design and evaluation, accuracy and human factors of human-computer interaction. 91 articles fulfilled all inclusion criteria. Some clear trends emerge. The Microsoft HoloLens increasingly dominates the field, with orthopaedic surgery being the most popular application (28.6\%). By far the most common surgical context is surgical guidance (n=58) and segmented preoperative models dominate visualisation (n = 40). Experiments mainly involve phantoms (n = 43) or system setup (n = 21), with patient case studies ranking third (n = 19), reflecting the comparative infancy of the field. Experiments cover issues from registration to perception with very different accuracy results. Human factors emerge as significant to OST-HMD utility. Some factors are addressed by the systems proposed, such as attention shift away from the surgical site and mental mapping of 2D images to 3D patient anatomy. Other persistent human factors remain or are caused by OST-HMD solutions, including ease of use, comfort and spatial perception issues. The significant upward trend in published articles is clear, but such devices are not yet established in the operating room and clinical studies showing benefit are lacking. A focused effort addressing technical registration and perceptual factors in the lab coupled with design that incorporates human factors considerations to solve clear clinical problems should ensure that the significant current research efforts will succeed.

1 Introduction

Despite decades of research and renewed interest in commercial OST-HMDs, AR surgical guidance has not become routine clinical practice, and evidence of clinical benefit remains lacking. This review surveys OST-AR surgery research and considers barriers to clinical adoption, especially human factors.

1 Introduction

  • AR surgical guidance has not translated into routine clinical practice despite decades of research and growing interest in commercial OST-HMDs.Its anticipated benefits include improved perception, ergonomics, hand-eye coordination, safety, reliability, repeatability, and surgical outcomes, but clinical benefit has not been demonstrated.Source: §1 · p. 1
  • The review categorizes OST-AR surgery research by application area, registration and validation methods, and visual content, while examining barriers to clinical adoption, particularly human factors.It describes common human-factor issues that may limit AR's applicability.Source: §1 · p. 1
  • Clinically successful AR should target specific procedural guidance needs, relevant information segmented from preoperative scans, and human factors in visualization and interaction.The review recommends that clinicians and engineers consider human factors early when developing OST-HMD AR projects.Source: §1 · p. 1

2 Background

OST-HMDs provide surgeons with direct, hands-free access to augmented information while preserving a view of reality, but registration, perception, and human-factor limitations constrain safe and effective surgical use. This review examines the devices, applications, experiments, accuracy, and human factors in the field.

2 Background

  • OST-HMDs preserve an unhindered real-world view and natural stereo vision without real-scene lag or resolution loss, but introduce registration errors, latency, complex calibration, and unnatural occlusion.Nearer virtual objects do not occlude real objects in the background.Source: §2 · p. 2

2.1 Commercial OST-HMD Devices

  • Commercial OST-HMDs followed technically difficult custom devices; Google Glass and HoloLens offered different form factors and capabilities, but none were designed specifically for surgery.HoloLens includes stereo vision, low-latency room mapping and head tracking, and hand-gesture interaction; the review notes only two included papers used custom devices.Source: §2.1 · p. 2

2.2 Safety

  • Condino et al. (2020) judged HoloLens unsuitable for high-precision manual tasks, while Carbone et al. (2020) deemed OST-HMDs unsuitable for surgical guidance and emphasized perceptual limits.Fida et al. (2018) concluded perceptual issues restrict use to simple virtual elements such as models, icons, or text.Source: §2.2 · p. 2
  • Human-factor risks include distraction, difficult gesture interaction, and cognitive overload; Cometti et al. (2018) found no impact on tested safety-critical functions but stressed user receptivity.Their study involved 12 volunteers performing manual and intellectual tasks with and without HoloLens for 90 minutes.Source: §2.2 · p. 2
  • Although some technical limits are being addressed, human-factor limitations remain the major hurdles to commercial success of OST-HMDs in surgery.The passage cites limited field of view and automatic eye-to-eye calibration as technical issues being addressed, for example in HoloLens 2.Source: §2.2 · p. 2

2.3 Efficacy

  • The review distinguishes registration or tracking accuracy from perceptual accuracy, while noting that large-scale clinical trials needed to demonstrate patient benefit are lacking.Its analysis covers devices, specialities, applications, procedures, visualisations, experiments, accuracy results, and human factors.Source: §2.3 · pp. 2–3

3 Methods

The review systematically examines original research on optical see-through head-mounted displays in surgical AR, focusing on their characteristics and human factors. It searches and screens literature from 2013–2020 and analyzes 91 eligible studies.

3.1 Literature search

  • The PRISMA-guided search used Google Scholar, covered 2013–2020, and was updated with a 2020-only search.The initial search was conducted on February 21, 2020, and the updated search on January 27, 2021.Source: §3.1 · p. 3

3.2 Other review papers

  • The review focuses specifically on OST-HMDs and critically analyzes their characteristics and human factors relevant to user acceptability.The authors note that existing reviews did not specifically cover OST-HMDs; related reviews commonly identify technological hurdles to acceptability and a lack of clinical validation.Source: §3.2 · p. 3

3.3 Literature analysis strategy

  • 91 studies met the review’s predefined eligibility criteria after screening records and assessing full texts.Studies had to be peer-reviewed original journal articles focused on OST-HMD surgical applications; full-text studies also had to investigate utility in surgical settings and not focus on optics or hardware design.Source: §3.3 · p. 4
  • A meta-analysis was not possible because the studies comprised small case series, subjective qualitative assessments, and no controlled randomized trials.The authors caution that publication bias could not be reduced and should be considered.Source: §3.3 · p. 4
  • Data extraction covered clinical setting, OST-HMD device, methods and visualizations, accuracy results, and human factors.Source: §3.3 · p. 4

4 Analysis of the literature search

The review analyzes 91 included articles from 2013–2020, covering OST-HMD devices and surgical applications. Publications increased from 2014, orthopaedic surgery was the largest specialty, and Microsoft HoloLens dominated device use.

4.1 Annual distribution of selected articles

  • No included articles met the criteria in 2013; publication numbers increased steadily from 2014 through 2020, a trend associated with releases of major OST-HMDs.The review notes Google Glass and Microsoft HoloLens as examples of major device releases associated with the rise.Source: §4.1 · p. 4

4.2 Surgical speciality

  • Orthopaedic surgery was the largest specialty, with 28.6% of publications (n = 26), while general, vascular, and neurosurgery grew more represented later in the survey period.The review suggests orthopaedic applications may be more feasible because proximity to bone allows rigid registration and somewhat lower accuracy than applications such as neurosurgery.Source: §4.2 · p. 4, Table 1 · p. 5

4.3 Optical see-through head-mounted displays (OST-HMDs)

  • Microsoft HoloLens dominated OST-HMD-assisted surgery studies, accounting for 66 articles; Google Glass ranked second with 8.HoloLens has led the field since its 2016 release, while interest in Google Glass declined from 2015 to 2017.Source: §4.3 · p. 5

5 Surgical application context

Surgical guidance was the dominant OST-HMD application context, with preoperative planning and training next most common. The review also describes varied guidance tasks and other uses spanning mentoring, anatomy assessment, and clinical consultation.

5.1 Surgical guidance

  • Surgical guidance includes registered pre- or intraoperative image overlays used to visualize and target anatomy, even without tracking, and spans multiple task-specific applications.General navigation overlays numbered 10; needle insertion 8; and tool placement and screw insertion 7 each.Source: §5.1 · pp. 5–7

5 Surgical application context

  • Guidance dominated the 91 included papers (n=58), followed by preoperative planning (n=12), training (n=11), teleconsultation and telementoring (n=5 each), and surgical anatomy assessment (n=4).Intraoperative imaging or record review, documentation, patient monitoring, and robot-assisted MIS preparation each had n=2.Source: §5 · p. 5, Figure 7 · p. 6

5.1 Surgical guidance

  • Other guidance tasks included catheter and K-wire insertion, endoscopy views, drill trajectories, saw navigation, and robot setup; safety depends on avoiding inaccurate guidance and visual clutter.Needle insertion studies reported clinically acceptable accuracy and fewer required CT scans; one dissection-guidance case was judged more reliable and less time-consuming than Doppler ultrasound.Source: §5.1 · pp. 7–8

5.2 Other surgical application contexts

  • Preoperative planning applications addressed coronary diagnosis interaction, communication of vascular MRI data, liver-resection planning, congenital heart repair, and nephron-sparing surgery.A study found that holographic liver anatomy visualizations improved users’ spatial understanding.Source: §5.2 · pp. 8–9
  • Training applications covered coronary diagnosis, hip-arthroplasty tool guidance, neurosurgical burr-hole localization, and transesophageal echocardiography; telementoring supported real-time shared views and annotations.Reported telementoring outcomes included improved information exchange, fewer focus shifts, reduced placement error, and greater trainee confidence.Source: §5.2 · p. 9
  • Teleconsultation provided on-demand colleague communication, including remote video and audio, tablet-based anatomical marking, and hands-free consultation during surgery.The reviewed examples used HoloLens or Google Glass across visceral, urological, reconstructive limb-salvage, and orthopaedic procedures.Source: §5.2 · pp. 9–10
  • Surgical anatomy assessment displayed preoperative 3D anatomy intraoperatively for decision-making rather than procedural guidance, while 2D images and records were viewed in the surgeon’s field of view.In one study, medical records were rated highly useful but patient-image review less useful; another enabled viewing and manipulation of X-ray and MRI images.Source: §5.2 · p. 10

6 AR visualisations

AR visualisations most often used preoperative models, while intraoperative models and images supported live guidance or access to patient data in the surgeon’s field of view. Other approaches—including holographic preoperative images, telementoring displays, numerical data, video, and documents—were less common.

6 AR visualisations

  • Preoperative models were the most common visualisation (n = 66), usually 3D reconstructions of CT or MRI anatomy, sometimes combined with planning components.Fig. 15 shows the distribution of articles by visualisation type.Source: §6 · p. 10, Figure 15 · p. 12
  • Preoperative models also included non-anatomical content, such as graphical annotations and interactive menus.Rojas-Muñoz et al. (2020a) used incision-line annotations and surgical-tool models; Condino et al. (2018) implemented toggle menus for orthopaedic training.Source: §6 · p. 10
  • Intraoperative model visualisations (n = 13) used updated live information, usually for guidance, including drill trajectories and needle position, orientation, and shape.Katić et al. (2015) used drill position, depth, and injury-avoidance warnings; Lin et al. (2018) examined needle visualisations.Source: §6 · p. 10
  • Live intraoperative images (n = 12) brought patient data into the surgeon’s field of view, including radiographs, endoscopy video, and registered C-arm X-rays.Examples include Deib et al. (2018), Qian et al. (2018), and Fotouhi et al. (2019a).Source: §6 · pp. 10, 12
  • Holographic preoperative images (n = 5) were proposed to bring images near the surgical site rather than requiring viewing on a separate monitor.Examples include Song et al. (2018), who combined 2D radiographs with guidance information, and Rynio et al. (2019), who displayed images, arterial measurements, and planning notes.Source: §6 · p. 12
  • Less common visualisations included live video streaming (n = 5), numerical data (n = 3), video communication planes (n = 2), pre-recorded video (n = 2), and documents (n = 1).Live streaming was mostly used for telementoring; numerical data supported decision-making or guidance, and pre-recorded video was explored for training.Source: §6 · p. 12

7 Validation of AR

All reviewed papers tested the usability and potential utility of OST-HMD-assisted surgery, but validation was concentrated in phantoms and system setup, and clinical effectiveness remains unproven.

7.1 Experimental setting

  • Phantoms (n = 43) dominate validation, followed by system-setup tests (n = 21) and patient case studies (n = 19), but clinical effectiveness and improved outcomes remain unestablished.System-setup tests examine intrinsic characteristics such as registration and calibration rather than realistic target anatomy.Source: Figure 16 · p. 13, §7.1 · p. 13
  • Other settings include simulator (n = 8), animal (n = 5), human-cadaver (n = 4), animal-cadaver (n = 1), and simulated-clinical-environment (n = 1) experiments.Animal studies enable testing under physiological conditions such as respiratory motion; cadavers permit testing on real anatomy without patient risk, while simulated clinical environments assess infrastructure and workflows rather than procedures.Source: §7.1 · p. 13

7.2 Evaluation methods

  • Quantitative experiments appear in 88 articles versus qualitative evaluations in 11; quantitative work measures accuracy or guidance, or uses numerically scored predetermined surveys.Examples include registration and calibration accuracy, tool positioning, and guide-wire placement.Source: §7.2 · p. 14
  • Qualitative evaluations are usually questionnaires describing observations that cannot be scored numerically, such as users’ reported benefits, limitations, and preferences after image-guided spine procedures.Deib et al. (2018) used this approach following a surgical image-guided spine procedure.Source: §7.2 · p. 14

8 Registration and tracking in surgical AR

Registration aligns a digital image with patient anatomy, while tracking maintains that alignment after motion; both technical error and user perception affect overall accuracy. Reported accuracy is difficult to compare because studies use varied experimental setups and metrics.

8.1 Manual alignment

  • Manual registration produced 4.39 ± 1.29 mm mean perceived holographic drift, versus 1.4 ± 0.67 mm with continuous marker tracking.Reflective AR displays improved accuracy from 30.2 ± 23.9 mm to 16.5 ± 11.0 mm; keyboard control was the most accurate of three HoloLens neuronavigation methods. Manual methods may not meet surgical-dissection guidance requirements, though orientation may aid spatial awareness and broader decision-making.Source: §8.1 · p. 14

8.2 Point-based registration

  • Point-based registration aligns corresponding anatomical or fiducial landmarks; target registration error (TRE) at the surgical target is more clinically relevant than fiducial registration error (FRE).Reported results include 0.809 ± 0.05 mm mean distance error and 1.038° ± 0.05° angular error; 2.24 mm average TRE; and 4.34 ± 0.709 mm FRE.Source: §8.2 · p. 15

8.3 Surface registration

  • Surface registration matches patient and image-space point clouds, typically after coarse manual or point-based alignment, and iterative closest point (ICP) is a commonly used method.HoloLens-generated surface meshes were found too coarse or biased for reliable registration and tracking.Source: §8.3 · p. 16
  • An improved ICP method achieved < 3 mm average target registration error on a dummy-head alignment test, while the HoloLens mesh and SLAM were unsuitable for registration and tracking.The HoloLens mesh was too coarse, with low vertex density and surface bias; SLAM also had uncertain realization.Source: §8.3 · p. 16

8.4 Other registration methods

  • The Fourier-based registration for transcatheter structural-heart guidance achieved 0.42 ± 0.02 mm accuracy.A HoloLens markerless facial-landmark method reported average positioning errors of 3.3 ± 2.3 mm on x, -4.5 ± 2.9 mm on y, and -9.3 ± 6.1 mm on z.Source: §8.4 · p. 17

8.5.1 Markerless tracking

  • Eleven papers used only HoloLens tracking, avoiding external devices and patient markers, but accuracy was user-dependent and might not suffice for some tasks.The method assumes the patient remains fixed in the operating room; marker attachment can offer a more accurate way to maintain alignment, but may be invasive.Source: §8.5.1 · p. 17

8.5.2 Tracking of markers using the OST-HMD device

  • OST-HMD camera tracking of surgical-field markers appeared in 20 papers, with Vuforia used for half of marker-based tracking through the device (9 papers).The camera’s position relative to the surgeon avoids extra registration, but its resolution and field of view may not suit tracking the surgical target.Source: §8.5.2 · p. 17

8.5.3 External tracking devices

  • NDI’s Polaris optical tracker appeared in 11 papers and its Aurora electromagnetic tracker in four; external tracking maintains alignment after patient or surgeon motion.Rigidly attaching markers to patients may be invasive, though it is used in commercial image-guidance systems and is described as probably the most accurate way to achieve and maintain alignment.Source: §8.5.3 · p. 17, §8.5 · p. 17

9 Human factors · 9.1 Human factors in AR · 9.2 HCI design considerations in OST-assisted surgery

Human factors—including perceptual, cognitive, and sensory-motor aspects—are central to OST-HMD acceptance and utility in surgery. The literature identifies challenges in evaluating AR usability and in designing OST-HMD interactions, especially calibration, but offers limited surgical research and no established generic human-factors guidelines.

9 Human factors

  • Human factors influence surgeons’ acceptance and the utility of OST-HMDs, alongside technological factors such as field-of-view size and video-streaming lag.They encompass perceptual, cognitive, and sensory-motor aspects of behavior that guide HCI design to optimize operator performance.Source: §9 · pp. 17–18
  • Generic human-factors guidelines for optical HCI have not been established, although suboptimal system design can cause predictable human mistakes in increasingly complex operating rooms.The authors argue that similar safety considerations apply to OST-HMD-assisted surgery.Source: §9 · p. 18

9.1 Human factors in AR

  • Human factors are a major hurdle in moving AR from laboratory prototypes to commercial products, prompting usability tests of perceptual needs and cognitive tasks.The passage describes low-level perceptual tests and task-based evaluations focused on the designed interface.Source: §9.1 · p. 18
  • Generalizable AR usability findings require carefully designed tasks; hardware limitations can make perceptual tasks difficult to generalize, especially when latency also affects cognition.The passage notes that application-specific cognitive tasks can be easier to design than general low-level perceptual tasks.Source: §9.1 · p. 18

9.2 HCI design considerations in OST-assisted surgery

  • Only a few studies address HCI design in OST-HMD-assisted surgery, and Tang et al. (2003) found that user factors substantially affect calibration error and produce different accuracy results across users.They recommend avoiding calibration based only on head movements, stabilizing the head by minimizing extrinsic body movement, and sequencing left- and right-eye data collection to avoid bias toward a dominant eye.Source: §9.2 · p. 18
  • Human factors affect both the HCI experience and calibration accuracy, making the accuracy of Guo et al.’s (2019) HoloLens online calibration method difficult to measure objectively.The passage also emphasizes the need for careful experimental design to obtain generalized results.Source: §9.2 · p. 18
  • Wearable HCI design should account for egocentric action and perception, because understanding human perception, cognition, and actions supports better cognitive assistance.The authors replace the terms “input” and “output” with “action” and “perception.”Source: §9.2 · p. 18

9.3 Human factors identification

The review identified user-related factors affecting acceptance, utility, and performance of surgery with or without OST-HMDs, including factors not explicitly called “human factors.” It grouped 34 factors by whether OST-HMD AR addresses them or they persist, and by three phases of user interaction.

9.3 Human factors identification

  • The review included user-related aspects described by authors as potential influences on acceptance, utility, and performance, even when articles did not explicitly call them human factors.Source: §9.3 · pp. 18–19
  • The review identified 34 human factors, grouping them as addressed by OST-HMD AR or persistent/inherent in proposed solutions, and by perception, cognitive processing, and control actions.These phases describe receiving device information, understanding and interpreting it, and acting on that interpretation.Source: §9.3 · p. 19, Table 4 · p. 19

9.4 Human factors of conventional surgery that are addressed by OST-HMD AR

The review groups conventional-surgery human factors targeted by OST-HMD AR into Information Perception, Cognitive Processing, and Control Actions. It highlights spatial awareness, attention switching, mental mapping, learning demands, surgical errors, and hand-eye coordination as key concerns.

9.4 Human factors of conventional surgery that are addressed by OST-HMD AR

  • Fig. 20 groups the identified limitations targeted by OST-HMD AR into Information Perception, Cognitive Processing, and Control Actions.The figure shows their distribution among 34 identified human factors; the supplied passages do not provide category-specific counts.Source: §9.4 · p. 19, Figure 20 · p. 19

9.4.1 Information Perception related Human Factors of conventional surgery

  • Spatial awareness was the leading Information Perception factor (n = 20): perceiving anatomy only in 2D can obscure spatial relationships and raise error risk.Examples include holographic endoscopy registered in the viewing frustum (Qian et al. (2018)) and spatially positioned C-arm images (Fotouhi et al. (2019a)).Source: §9.4.1 · pp. 19–20

9.4.2 Cognitive Processing related Human Factors of conventional surgery

  • Attention switching was the most frequently identified factor overall (n = 41), as looking between the surgical site and a separate monitor can cause distraction and disrupt continuity.During image-based navigation, repeated shifts while manipulating tools may also impair hand-eye coordination and concentration (Wang et al. (2016); Chen et al. (2015)).Source: §9.4.2 · p. 20
  • Mental mapping of 2D images to the 3D patient or surgical scene appeared in 24 articles and can be error-prone under high mental workload (Andress et al. (2018)).The issue is especially relevant to precise tool or implant placement under C-arm fluoroscopy.Source: §9.4.2 · p. 20
  • A steep learning curve was identified in 20 articles; OST-HMD AR was proposed to reduce learning demands in needle guidance (Lin et al. (2018)).Challenges included reconstructing 3D information from 2D images, identifying the needle tip and target despite artifacts, and coordinating hand and eye across screen and patient.Source: §9.4.2 · p. 20

9.4.3 Control Action related Human Factors of conventional surgery

  • Surgical error risk was addressed in 23 articles, including concerns about divided attention to the scene and monitor and errors such as root-canal perforation (El-Hariri et al. (2018); Song et al. (2018)).Source: §9.4.3 · p. 20
  • Unfamiliar hand-eye coordination was addressed in 18 articles; causes include screen-based guidance that is misaligned with the surgical site or hides the surgeon’s hands.Examples include registered holographic endoscopy for robotic laparoscopic surgery (Qian et al. (2018)) and indirect fluoroscopic guidance in percutaneous spine procedures (Deib et al. (2018)).Source: §9.4.3 · p. 20

9.5 Persistent human factors of proposed OST-HMD solutions

Proposed OST-HMD solutions introduce persistent human factors involving information perception, cognitive processing, and control actions. The review describes both user-acceptance challenges and ways users respond to new visualisation and interaction options.

9.5.1 Information Perception related Human Factors of AR-assisted surgery

  • Comfort was reported in 15 articles; surgeons described good long-term wear ergonomics with Movierio BT-200 glasses, while HoloLens weight negatively affected posture and comfort.These contrasting reports show that comfort can vary across proposed OST-HMD solutions.Source: §9.5.1 · p. 21
  • Spatial perception appeared in 14 articles; simulator users found relationships easy to perceive, engineers rated this slightly higher than clinicians, and learners rated HoloLens above lectures and computer modules on three measures.The three measures were learning effectiveness, conveying anatomical spatial relationships, and learner engagement and motivation.Source: §9.5.1 · p. 21
  • In a phantom K-wire insertion experiment, HoloLens rendering at a fixed focal distance of around 2 m impaired depth perception and interaction between real and virtual objects.Missing or impaired depth perception was reported as a limitation in seven articles.Source: §9.5.1 · p. 21

9.5.2 Cognitive Processing related Human Factors of AR-assisted surgery

  • Ease and intuitiveness appeared in 20 articles; users initially preferred a conventional 2D-like mode but shifted to a mixed-reality mode after familiarisation, while some preferred gestures because voice interfered with patient communication.The voice-versus-gesture finding came from a Google Glass personal assistant experiment.Source: §9.5.2 · pp. 21–22
  • Usefulness appeared in seven articles; in one patient case study, 74% of surgeons rated Google Glass-assisted access to preoperative CT scans as high or very high.The evaluation involved seven surgeons and reported 10 procedures totalling 31 procedures.Source: §9.5.2 · p. 22

9.5.3 Control Action related Human Factors of AR-assisted surgery

  • Preferred visualisation mode was the most frequently reported control-action factor, appearing in six articles; one robotic-surgery assistant could choose between a holographic endoscopy monitor and a view-frustum-registered display.The example is from Qian et al. (2018).Source: §9.5.3 · p. 22

10 Discussion

OST-HMD surgical research is expanding, but clinical use and demonstrated utility remain limited. Registration, human factors, and task-specific evidence are central challenges in translating these systems into practice.

10 Discussion

  • Orthopaedics accounts for 30.16% of reviewed applications, mainly intraoperative guidance, possibly reflecting its rigid bony structures and established guidance practices.Source: §10 · p. 22
  • Image guidance dominates across specialities, but reported accuracy varies with experiments and measures, and no general registration solution exists.Guidance commonly aligns a preoperative segmented model with the operative view; generating these models requires segmentation in the surgical planning workflow, a process outside this review’s scope.Source: §10 · p. 22

10.1 Human factor classification

  • OST-HMDs aim to reduce attention shifts from the surgical site, but ease and intuitiveness remain persistent concerns, while devices can impair hand-eye coordination and increase cognitive load.The review notes that these factors may increase rather than reduce surgical-error risk.Source: §10 · p. 22, §10.1 · p. 22

10.2 Potential machine learning applications

  • None of the 91 selected articles applied machine learning, although OST-HMD video, gesture, eye-tracking, and surface-mesh data could support future algorithms.The authors identify relevant user-data databases as a fundamental step for advancing machine-learning research in surgery.Source: §10.2 · p. 24

10.3 Conclusions

  • OST-HMD surgical publications and application diversity have risen, yet few systems have been used clinically and demonstrated utility remains rare.The Microsoft HoloLens boosted mixed-reality surgical research from 2017 onward.Source: §10.3 · p. 24
  • Directly overlaid AR can cause inattention blindness even with perfect registration; nearby, side-by-side guidance could preserve an oriented model view without obscuring the real scene.Dilley et al. (2019) compared direct AR with nearby unregistered guidance in a screen-based simulation.Source: §10.3 · p. 24
  • Research should target procedure-specific moments affecting patient outcomes, test whether tailored visualisations improve performance, and evaluate OST-HMDs’ potential for surgical learning.Viewing anatomy and pathology in situ may improve novice surgeons’ spatial understanding and reduce the learning curve; Louis et al. (2020) demonstrated improved learning with AR under high-fidelity conditions, motivating OST-HMD learning studies.Source: §10.3 · pp. 24–25

Appendix A Papers summary table

The appendix table catalogs AR visualizations, experiments, and reported accuracy across reviewed studies, spanning planning, surgical guidance, and training applications.

Appendix A Papers summary table

  • Table A1 summarizes AR visualizations, experiments, and reported accuracy for the final 91 articles used in quantitative synthesis.Source: Table A1 · p. 31, Appendix A · p. 31
  • The table’s experiment categories include system setup, phantom, human or animal cadaver, simulator, simulated clinical environment, and patient case studies.Source: Appendix A · p. 31
  • Accuracy entries include El-Hariri et al.’s fiducial-marker RMSE (x: 3.22 mm, y: 22.46 mm, z: 28.30 mm) and Zhou et al.’s average needle location error (PE: 0.957 mm; AE: 2.416 mm).Source: Appendix A · pp. 32–33
  • For pediatric elbow fractures, Laguna et al. used AR models for presurgical planning and orthopedic surgeons assessed the 3D models.Source: Appendix A · p. 33
  • Dallas-Orr et al. presented a 3D spine model with a vascular overlay, measured using circumference and angle tools in standard-of-care PACS software.Source: Appendix A · p. 33
  • Other entries describe a digital skull-anatomy session using HoloHuman and bladder-catheter placement practice with a male training model.Source: Appendix A · p. 33
  • Luzon et al. evaluated registration and needle placement with 3D anatomy models, reporting an x-axis target error distance of 2.9757 ± 1.33396 mm.The table fragment does not provide the remaining axis values.Source: Appendix A · p. 33
  • 1.74 ± 0.86 mm total target overlay error across 336 targets and 0.41 ± 0.23° needle overlay angle were reported for Yang et al.’s simulations.The experiments involved simulated portal-vein puncture and TIPS in dogs, plus a liver-phantom study.Source: Appendix A · p. 33
Loading 2202.04141v1…