Source-linked AI summary
The HoloLens in Medicine: A systematic Review and Taxonomy
Christina Gsaxner, Jianning Li, Antonio Pepe, Yuan Jin, Jens Kleesiek, Dieter Schmalstieg, Jan Egger
TL;DR
This systematic review examines how the first-generation HoloLens has been used in medicine and what evidence supports its clinical value. It finds few clear advantages over current practice, with promise in some procedures but continuing technical and practical constraints.
Problem
Across investigated medical specialties and applications, few studies have clinically demonstrated clear advantages of HoloLens-based systems over the current state of the art.
Source: §10 · p. 21Method
The review classified each study by intended HoloLens user and organized extracted publication information according to a taxonomy.
Source: §3.2 · p. 4Results
Few studies clinically demonstrated clear advantages over the current state of the art; tracking errors are generally too high for high-precision applications, while some procedures not requiring sub-millimeter precision appear promising.
Source: §10 · p. 21Takeaways & Limitations
Establishing AR in clinical practice requires greater efforts in precision, reliability, usability, workflow, and perception.
Source: §10 · p. 21The HoloLens is not yet suitable for extended everyday use because it is relatively expensive, has short battery life, and is bulky.
Source: §6.3.2 · p. 11
Abstract
from arXiv · showhide
The HoloLens (Microsoft Corp., Redmond, WA), a head-worn, optically see-through augmented reality display, is the main player in the recent boost in medical augmented reality research. In medical settings, the HoloLens enables the physician to obtain immediate insight into patient information, directly overlaid with their view of the clinical scenario, the medical student to gain a better understanding of complex anatomies or procedures, and even the patient to execute therapeutic tasks with improved, immersive guidance. In this systematic review, we provide a comprehensive overview of the usage of the first-generation HoloLens within the medical domain, from its release in March 2016, until the year of 2021, were attention is shifting towards it's successor, the HoloLens 2. We identified 171 relevant publications through a systematic search of the PubMed and Scopus databases. We analyze these publications in regard to their intended use case, technical methodology for registration and tracking, data sources, visualization as well as validation and evaluation. We find that, although the feasibility of using the HoloLens in various medical scenarios has been shown, increased efforts in the areas of precision, reliability, usability, workflow and perception are necessary to establish AR in clinical practice.
1 Introduction
The introduction presents the HoloLens as a way to overlay clinical information and guidance in medical settings, then asks how much these envisioned uses have been realized. The review surveys first-generation HoloLens medical research from 2016 to 2021 and analyzes its applications, technical methods, data, visualization, and evaluation.
1 Introduction
- The HoloLens could provide physicians with patient information in their field of view, support medical students’ understanding of anatomy and procedures, and guide patients through therapeutic tasks.The introduction also describes potential uses in remote expertise, rehabilitation, and intervention practice.Source: §1 · p. 2
- The introduction asks how much the HoloLens has contributed to realizing these potential medical uses.It frames this question against the HoloLens 1’s role as a fully untethered wearable AR device.Source: §1 · p. 2
- The review covers first-generation HoloLens medical research from 2016 to 2021, analyzing 171 publications by use case, registration and tracking, data and visualization, and evaluation and validation.The authors also identify gaps, challenges, and limitations.Source: §1 · p. 2
2 Background
The background introduces AR and medical registration and tracking, then traces the return to optical see-through headsets after earlier display constraints. It describes the HoloLens as a self-contained device with onboard sensing, tracking, and interaction capabilities.
2.1 Augmented reality
- AR combines real-world views with virtual content; medical registration aligns that content with the patient, while tracking maintains the viewing camera’s position and orientation.The review describes outside-in tracking as using external sensors and inside-out tracking as using device-integrated sensors.Source: §2.1 · p. 2
- Head-mounted displays align content with the wearer’s gaze and leave the hands free, but early systems often required bulky, wired computing and tracking infrastructure.Research shifted toward world-localized displays between 2011 and 2017, before the HoloLens 1 renewed attention to optical see-through displays.Source: §2.1 · p. 2
- Marker-based tracking uses artificial features such as fiducials or IR emitters, while marker-less methods use natural features, SLAM, or models fitted to real-world objects.The review notes that patient skin-surface models from medical imaging can suit model-based approaches.Source: §2.1 · p. 3
2.2 The HoloLens
- The HoloLens 1 is an untethered optical see-through headset released in 2016, with onboard sensors, spatial mapping, inside-out tracking, and gesture or voice interaction.Its display has a 34-degree diagonal field of view, and its specified active battery life is 2–3 hours.Source: §2.2 · p. 3
3 Methodology
The methodology describes a PRISMA-guided search and screening process, followed by extraction and classification using a review-specific taxonomy. It positions this review among prior AR reviews and focuses on challenges that remain relevant to newer optical see-through headsets.
3.1 Search strategy and selection process
- The PRISMA-guided search covered PubMed and Scopus for HoloLens and medical terms, restricted publications to 2016–2021, and screened English-language, peer-reviewed original research focused on human medical use.The search yielded 975 records plus 18 known records; screening left 171 studies for final analysis.Source: §3.1 · pp. 3–4, Figure 1 · p. 4
3.2 Data extraction and taxonomy
- The authors extracted publication and specialty information, then classified studies by intended user and application, tracking approach, data source and type, acquisition time, visualization, and evaluation.The taxonomy also covers validation; Figure 2 summarizes its dimensions.Source: §3.2 · p. 4, Figure 2 · p. 5
3.3 Related reviews
- Prior reviews repeatedly noted limited clinical validation, tracking and rendering constraints, and workflow-related usability challenges; this review focuses on challenges relevant to newer OST-HMDs.The authors state that HoloLens self-tracking, interface support, and rendering make some earlier challenges obsolete, while newer devices retain relevant concerns.Source: §3.3 · pp. 4–5
4 Publications per year
The reviewed publication count increased after the HoloLens’s 2016 release, peaked in 2020, and declined in 2021. The authors suggest the newer HoloLens 2 may have contributed to the final-year decline.
4 Publications per year
- No medical-domain HoloLens publications appeared in 2016, although the device became available that year.It launched in North America in March and worldwide in October.Source: §4 · p. 5
- Publication counts across categories rose steadily after 2016 and reached their highest level in 2020.Source: §4 · p. 5
- Publication counts fell in 2021, a decline the authors suggest was likely related to researchers shifting attention to the HoloLens 2.Source: §4 · p. 5
5 Medical fields of applications
HoloLens applications spanned 21 medical fields, with orthopedic surgery and neurosurgery most frequently represented. Some studies did not specify a field, while patient-focused applications tended to target specialties where cooperation and motivation affect treatment outcomes.
5 Medical fields of applications
- Orthopedic surgery (18) and neurosurgery (14) were the most frequently supported fields, especially among physician-targeted applications.The review grouped applications into 21 medical fields.Source: §5 · p. 5
- Fifteen publications named no specific medical field, and eleven targeted surgical procedures generally, so their concepts could span specialties.Patient-focused applications tended to address specialties such as neurology and kinesiology, where patient cooperation and motivation can affect treatment outcomes.Source: §5 · p. 6
6 Use cases
The review organizes HoloLens use cases by intended user and application area. Most publications targeted clinical professionals, followed by medical students and patients.
6 Use cases
- Clinical professionals were the dominant target: 128 of 171 studies, almost 75%, addressed tasks such as diagnosis, treatment planning, or treatment execution.Medical students accounted for 24 studies and patients for 19.Source: §6 · p. 6
- The review groups applications into three user categories and subcategories based on application area; Figure 4 covers 21 medical fields, including nonspecific applications usable across disciplines.“Nonspecific” denotes applications whose authors did not name one specific medical area.Source: Figure 5 · p. 6, Figure 4 · p. 6
6.1 Physician-centered applications of the HoloLens
Physician-centered HoloLens applications range from displaying medical data to image-guided interventions and surgical navigation. These use cases increase in technical complexity as registration and tracking demands grow.
6.1 Physician-centered applications of the HoloLens
- The review classifies physician-centered applications by increasing complexity: data display, image-guided intervention, then surgical navigation.Navigation adds tool tracking to patient and headset tracking, producing the highest accuracy and reliability demands.Source: §6.1 · pp. 6–7
6.1.1 Data display
- Data-display applications use the HoloLens as an immersive display for medical images or records, without requiring alignment between the data and physical space.The review identified 33 publications, most using pre-intervention 3D imaging; world-anchored content can be examined from different perspectives, and touchless interaction may suit sterile settings.Source: §6.1.1 · p. 7
- Telemedicine applications use headset video, voice, or annotations to connect remote experts with local staff during interventions without requiring users to look away from the patient.Examples included e-consulting for rural and remote hospitals and operating-room telemonitoring.Source: §6.1.1 · p. 7
6.1.2 Image-guided interventions
- Image-guided intervention was the most common physician-centered use case, using aligned imaging to show anatomy beneath skin or obstructive structures.Applications included minimally invasive procedures and exposed surgical sites, where overlays could show critical structures or planning information.Source: §6.1.2 · p. 7
- Convincing X-ray visualization requires accurate image-to-patient registration, while display calibration and stable HoloLens self-tracking also matter.Many reviewed works relied on manual alignment; others developed or compared calibration and registration methods.Source: §6.1.2 · p. 7
- Live imaging applications enhance procedures such as laparoscopy, endoscopy, fluoroscopy, or ultrasound by delivering intra-operative data to the headset in real time.Studies evaluated intra-operative X-ray, endoscopy, ultrasound, electro-anatomic mapping, and MRI.Source: §6.1.2 · pp. 7–8
6.1.3 Surgical navigation
- Surgical navigation fuses guidance with the operative site but requires real-time tracking of instruments relative to patient anatomy, in addition to image-to-patient registration.The review found 27 HoloLens navigation studies; the hardware can struggle with millimeter or sub-millimeter accuracy and reliability demands, while its slimmer form factor suits less critical procedures usually performed without navigation.Source: §6.1.3 · p. 8
6.2 Applications of the HoloLens for medical students
The review identified 24 HoloLens publications supporting medical and healthcare students, covering interventional or surgical training and anatomy learning. Training studies used hybrid or virtual scenarios, while anatomy work explored 3D visualization.
6.2 Applications of the HoloLens for medical students
- The review identified 24 student-support publications, divided between interventional or surgical training and anatomy learning.The authors note that research was less numerous than might be expected given available commercial solutions.Source: §6.2 · p. 9
6.2.1 Interventional and surgical training
- Fifteen studies addressed interventional or surgical training, including hybrid simulators with added guidance or direct feedback and fully virtual scenarios.Applications included orthopedic and emergency procedures, laparoscopy, ultrasound examinations, and urological procedures.Source: §6.2.1 · p. 10
6.2.2 Anatomy learning
- Anatomy-learning studies used HoloLens 3D visualizations of anatomical models and also examined microscopic anatomy.The review describes AR as able to annotate physical models or cadavers with digital information.Source: §6.2.2 · p. 10
6.3 Patient-focused applications of the HoloLens
The review groups patient-focused HoloLens applications into training and education, assistance and monitoring, and assessment and diagnosis. Assessment studies report reliable sensor-derived measurements, while everyday use remains constrained by device cost, battery life, and bulk.
6.3 Patient-focused applications of the HoloLens
- The 19 patient-focused systems fall into training and education, assistance and monitoring, and assessment and diagnosis.The review notes that designing for patients must account for age, varying affinity for new technology, and anxiety around medical treatment.Source: §6.3 · p. 10
6.3.1 Patient training and education
- Immersive training and pre-interventional education have been associated with improved patient engagement and satisfaction, while HoloLens systems also support cognitive-disorder and prosthesis training.The review describes HoloLens-based explanations intended to make information before surgery more comprehensible and imaginable.Source: §6.3.1 · p. 10
6.3.2 Assistance and monitoring
- Patient assistance applications span navigation, daily activities, care-plan adherence, and text editing, but cost, short battery life, and bulk limit extended everyday use.The review identifies mHealth applications as a promising use of head-worn AR, including scenarios outside clinical environments.Source: §6.3.2 · pp. 10–11
6.3.3 Assessment and diagnosis
- All reviewed sensor-based assessment studies showed measurement reliability, and the HoloLens can guide and track mobility tests and measure gait parameters.The review notes that screening use is feasible during the confined assessment period, with potential ergonomic and economic benefits from combining monitoring and instructions on one device.Source: §6.3.3 · p. 11
7 Registration and object tracking with the HoloLens
The review examines registration between virtual and physical content and object tracking across medical HoloLens studies. Inside-out marker-based methods are common, while manual, inside-out, and outside-in approaches each carry distinct workflow or accuracy trade-offs.
7 Registration and object tracking with the HoloLens
- Of 99 studies establishing registration, 31 also integrated object tracking; the most common approach was inside-out marker-based tracking, followed by manual alignment.Figure 9 presents the frequency of registration and tracking methods identified in the review.Source: §7 · p. 12, Figure 9 · p. 12
7.1 Manual registration
- Manual registration aligns virtual items by position, orientation, and scale; 32 studies used it, but alignment can be slow and must be repeated if the patient moves.Landmark-based alignment can reduce the burden, but coarse spatial maps and absent haptic feedback may undermine accuracy and reliability.Source: §7.1 · p. 12
7.2 Inside-out methods
- Inside-out methods work in unprepared environments without specialized external hardware, but marker-based registration can be inaccurate and sensitive to viewing conditions.Marker-based registration was the most common technique, used in 42 studies; patient-attached markers may require pre-interventional imaging, while fiducial errors can range from millimeters to centimeters.Source: §7.2 · pp. 12–13
- Inside-out marker-less work includes surface, landmark, image, and point-cloud registration, with Research Mode enabling use of depth data in some methods.A point-cloud matching pipeline used 3D features from time-of-flight depth alone; combining facial landmarks with depth data was reported to slightly improve accuracy.Source: §7.2 · p. 14
7.3 Outside-in methods
- Outside-in systems can provide high reference precision, often below one millimeter, but calibration can disrupt workflow and optical marker tracking requires line of sight.Electromagnetic tracking is another outside-in option, though metallic materials common in clinical spaces can interfere with it.Source: §7.3 · p. 14
8 Data and visualization
The review categorizes HoloLens visualizations by data source, type, acquisition time, and presentation. Medical imaging is prominent, while planning data, tracked objects, and non-medical sources also contribute to the reviewed applications.
8 Data and visualization
- 3D medical imaging is by far the most common visualization data source, and reviewed studies often used more than one source or data type.Figure 10 shows the frequency of data sources; the review notes that its categories can have multiple mentions per study.Source: §8 · pp. 14–15, Figure 10 · p. 15
8.1 Acquisition time
- Pre-interventional data accounts for 206 examples, compared with 58 intra-interventional data sources collected and streamed at run time.Intra-interventional workflows require a device connection to the raw data source and automatic real-time processing.Source: §8.1 · p. 15
8.2 Medical data
- 110 reviewed papers used 3D medical images, primarily CT/CTA (89) and MRI (38); slice rendering appeared in 29 systems, while only six attempted volume rendering.Surface rendering is computationally efficient but requires segmentation; direct volume rendering avoids surface extraction but is demanding for mobile hardware.Source: §8.2 · p. 15
- 2D medical imaging appeared in 20 studies, with 15 supporting run-time acquisition; other medical inputs included planning data in 29 publications, tracked-object coordinates, and biosignals or records.Planning information such as access points, trajectories, resection margins, and implant targets is commonly represented as geometric primitives relative to the target anatomy.Source: §8.2 · pp. 15–16
8.3 Non-medical data
- Eight studies used patient surface scans and 11 used artist-created anatomical models; both sources were deployed pre-interventionally, but surface scans do not show underlying anatomy.The review also describes non-medical 3D data for serious games and procedure simulations.Source: §8.3 · p. 16
- Eleven studies visualized non-medical 2D videos or documents, while six explored other inputs such as user motion, gaze, or external IMU data.The reported uses included measuring gait, guiding the wearer, and training limb-prosthesis control.Source: §8.3 · p. 16
9 Evaluation of medical HoloLens applications
Evaluation of medical HoloLens applications spans proof-of-concept, laboratory, and relevant-environment studies, but heterogeneous metrics and barriers to rigorous clinical evaluation limit comparability. Findings include mixed clinical task outcomes, variable registration accuracy, and generally favorable but qualified user assessments.
9 Evaluation of medical HoloLens applications
- Evaluation is difficult to standardize because user perception varies, benchmarks and protocols are absent, and institutional approvals can impede clinical trials and quantitative data collection.The review therefore found diverse metrics across distinct experimental scenarios.Source: §9 · p. 16
9.1 Evaluation scenario
- Among reviewed AR studies, 89 evaluated systems in relevant environments, versus 47 laboratory records and 35 proof-of-concept papers; relevant-environment studies were the largest category.Relevant-environment studies evaluated systems where they should be implemented, involving intended target-group users; questionnaires were collected most of the time. Laboratory studies typically focused on technical aspects and reported quantitative measurements.Source: §9.1 · p. 17
9.2 Quantitative metrics
- The HoloLens was judged suitable for displaying medical data with an appropriate software framework, including in safety-critical environments, but battery life limited use to around two hours.Source: §9.2 · p. 17
- Compared with other optical see-through headsets, HoloLens was favored for contrast, frame rate, and perceived task load in one comparison, and by students for untethered use and model stability in another.The latter comparison also criticized its smaller field of view and more complicated operation.Source: §9.2 · p. 17
- TRE was used in 22 studies, with reported averages ranging from just above 1 mm to 40 mm; image-fiducial accuracy also depended on lighting, viewing angle, and movement.The review notes that TRE measurements themselves can be subjective and error-prone, contributing to variability.Source: §9.2 · pp. 17, 20
- AR-supported procedures were sometimes slightly less accurate than conventional image-guided procedures but faster, while other studies reported favorable accuracy or less radiation use.Comparisons with freehand, non-guided procedures reported improved accuracy and more successful task completions; task-time and success-rate findings were not uniformly favorable.Source: §9.2 · p. 20
9.3 Qualitative metrics
- Reported questionnaire outcomes generally favored HoloLens and AR, but users also cited limited field of view, short battery life, discomfort, registration drift, and depth-perception problems.Comparative NASA-TLX findings were mixed, and only two studies reported overall SUS scores, both above average.Source: §9.3 · p. 21
10 Conclusion and outlook
The review finds that first-generation HoloLens applications have been explored across medical specialties, but clear clinical advantages over current state-of-the-art are rare. Clinical adoption still faces technical, perceptual, and evaluation challenges, while selected lower-precision scenarios and future hardware remain promising.
10 Conclusion and outlook
- Although feasibility has been suggested across medical scenarios, few HoloLens systems have clinically demonstrated clear advantages over current state-of-the-art.The authors conclude that medical AR research remains in its early stages, with substantial future work remaining.Source: §10 · pp. 21, 25
- For high-precision image-guided applications, HoloLens registration and tracking errors are generally too high, while some procedures not requiring sub-millimeter precision appear promising.The authors identify ablations, ventriculostomy, and certain orthopedic interventions as examples; automatic and accurate inside-out registration and tracking are needed to keep setup workflow.Source: §10 · p. 21
- Students are the second most common target group; HoloLens can enhance simulator training, while anatomy-learning effects versus conventional methods seem small.Studies report improved engagement and motivation, but more innovative, interactive, dynamic visualizations have not been explored in depth.Source: §10 · p. 21
- Patients are the least frequent target group, and only a small number of assistance and monitoring applications have been explored in selected therapy and diagnosis contexts.The review notes restricted possible usage for many patient applications.Source: §10 · pp. 21, 25
- Image fiducials are the most common inside-out registration approach and can provide reliable, acceptable accuracy in controlled settings, but line-of-sight and viewing-condition sensitivities may impede clinical adoption.Marker-less inside-out registration and instrument tracking remain technically limited, though deep-learning approaches for instrument tracking are gaining traction.Source: §10 · p. 25
- Most reviewed systems display pre-intervention CT or CTA as surface-rendered 3D data, while depth-perception problems can impair perceived accuracy and guidance.Few reports address visualization strategies that overcome these issues, and the HoloLens's additive-only display constrains the applicability of some depth-perception methods.Source: §10 · p. 25
- Evaluation in relevant settings is common, but varied experimental setups and measures and the lack of standardized protocols make clinical validation difficult.The authors call for larger comparative studies and collaboration among researchers, universities, clinicians, and patients to move medical AR beyond controlled laboratory settings.Source: Table 7 · p. 22, §10 · p. 25
- The authors expect medical AR research to continue with newer hardware, while specialized medical displays may address limitations of current commercial devices.They suggest improved hardware could facilitate on-device deep-learning models for recognition, tracking, and scene understanding.Source: §10 · p. 25