Seeing through the Eyes of AI: Situated Explainability in Augmented Reality

1 INTRODUCTION

• AI methods can make errors and reflect dataset bias, yet their decisions often come with little or no explanation.The paper notes that this lack of transparency can contribute to user errors and reduce confidence in the system.Source: §1 · p. 1
• CAM heatmaps highlight class-relevant regions, helping users assess whether a visual model attends to relevant features.The paper notes CAM’s uses in identifying biases, spurious correlations, misclassifications, and other behaviors.Source: §1 · p. 1
• Conventional CAM displays often use previously captured images or a separate 2D view, requiring users to map displayed locations back to physical objects.AR can spatially register explanations to their physical referents, enabling inspection in the model’s operating context.Source: §1 · p. 2
• Situated explainability visualizes AI explanations in the physical world, registering them to model inputs and synchronizing them with decisions at runtime.The paper presents a first implementation for immersive exploration and reports a user study with participants who had AI backgrounds; participants found it more engaging but reported higher cognitive workload and preferred it to traditional tools.Source: §1 · p. 2

2 RELATED WORK

• Physical-AI applications include AR guidance and object search; foundation models offer flexibility, whereas standard detectors are optimized for faster, lower-latency deployment.The paper describes foundation models as larger, slower, and more complex to deploy in their current form.Source: §2 · p. 2

2.2 Explainability in AR and VR

• Existing AR explainability research is still early, and prior work has explored spatially coupling explanations with the objects, regions, and actions they address.The paper embeds fine-grained saliency maps in the physical environment rather than using a separate 2D or virtual interface.Source: §2 · p. 2, §2.2 · p. 2

2.1 Explainability visualizations

• Post-hoc visual methods such as SHAP and GradCAM explain trained models, while interactive systems primarily support offline analysis of pre-recorded data on desktop displays.Natural-language explanations have also been used, but LLM-generated accounts can be inaccurate.Source: §2.1 · p. 2

2.2 Explainability in AR and VR

• AR XAI work includes textual frameworks and LLM-generated explanations, while De Heuvel et al. [8] visualize whole-object attributions for robot navigation in virtual reality.The paper contrasts its fine-grained visual explanations in users’ real physical environments with these approaches.Source: §2.2 · p. 2
• Figure 2 sketches situated-explainability applications in chip inspection and physiotherapy, including a possibly incorrect association between a nearby battery and a missing-battery error.Its motion example highlights parts of a 3D trajectory that influenced an “incorrect” classification.Source: Figure 2 · p. 3

2.3 AR debugging and data collection

• AR debugging tools such as ARGUS focus on performance statistics or data collection, while ARCADE replays recorded captures to evaluate CV models.The paper instead focuses on real-time understanding of model decisions through spatially registered visual explanations, which can also support informed data collection.Source: §2.3 · p. 3

3 SITUATED EXPLAINABILITY

• Explanations are generated in real time and registered to the objects they explain, allowing interactive model exploration and additional data collection when models underperform.The paper frames this as bringing methods originally intended for offline inspection into the user’s environment.Source: §3 · p. 3
• In industrial inspection, 3D CAM maps could help technicians check whether defect detectors rely on fault regions rather than lighting or texture variations.Technicians can inspect an object from multiple viewpoints to check detection stability.Source: §3 · p. 3
• In retail, visualizations could reveal whether product recommendations rely on product features or background and packaging biases.Users could adjust their view to obtain different recommendations.Source: §3 · p. 3
• During model development, live saliency visualizations could help engineers compare architectures and check whether networks use relevant features or dataset shortcuts.AR lets them vary live conditions in a controlled test environment.Source: §3 · p. 3
• For motion classification, CAM could visualize which parts of a time-varying trajectory are discriminative.The paper gives a physiotherapy tutorial as an example application.Source: §3 · p. 3
• In AR re-localization, in-situ explanations could highlight scene features used by the model for users to inspect.The proposed scenario lets users move around and check whether relevant regions are treated as important.Source: §3 · p. 3
• Live AR explanations could support quicker decisions about how to proceed when users discover learned flaws, unlike traditional explanations on pre-recorded data.This is presented as a potential advantage of live integration.Source: §3 · p. 3
• Users could occlude or move relevant scene parts and observe changes in model accuracy, recovery, and explanation visualizations.The paper presents this as a way to inspect model sensitivity.Source: §3 · p. 3

4 APPLICATION

• The proof-of-concept processes headset images with a neural network and explainability algorithm, displays results live in AR, and supports offline analysis and later data capture for retraining.The pipeline can also record data for offline use.Source: §4 · p. 3, Figure 4 · p. 4

4.1 Live inspection mode

• LayerCAM heatmaps are registered at object locations alongside detections, exposing class-relevant regions and features too weak to trigger full detections.Heatmaps are computed at four frames per second without stalling the headset’s AR experience.Source: §4.1 · p. 4
• Users can place 3D ground-truth boxes and class labels on objects to identify misclassifications, false positives, and false negatives during live inference.The labels can also be used to automatically record ground-truth data.Source: §4.1 · p. 4, Figure 5 · p. 4

4.2 Offline analysis mode

• Offline maps accumulate live heatmap point clouds on the workspace and relate activations to camera viewpoints, whose clustered poses and directional arrows help inspect problematic views.Camera poses within 15 centimeters and 15 degrees are clustered together.Source: §4.2 · p. 4
• AR hints highlight outlier areas with false positives or false negatives, while intersection-over-union checks and session statistics summarize detection performance.The statistics include frame, false-positive, false-negative, and true-positive counts.Source: §4.2 · pp. 4–5

4.3 Data capture

• Projected 3D labels are tightened with Segment Anything masks to produce 2D object-detection annotations compatible with YOLO-style training.The batch processing requires no user intervention.Source: §4.3 · p. 5

4.4 Implementation details

• The evaluated implementation uses a Meta Quest 3, Unity3D, and a pretrained YOLO model, with class-loss-based explanations targeting one class at a time.The implementation uses a low detection-confidence threshold of 0.3 to capture more information.Source: §4.4 · p. 5

5.1 Participants and study design

• The exploratory study recruited six AI-experienced users for cluttered-desk and pantry-detector tasks, with UI familiarization, think-aloud use, and follow-up interviews.All participants had AR experience, and all but one had used explainability tools.Source: §5.1 · p. 5

5.2 Results

• Participants valued moving around real objects with real-time, object-aligned feedback; live heatmaps prompted closer inspection and surfaced unexpected detector failures.Participants also described the visualizations as helping them understand detector behavior.Source: §5.2 · p. 6
• Offline camera markers and arrows helped users track covered viewpoints, and inspecting heatmaps naturally suggested recording training data.Participants also found the application easier to interpret than raw numbers.Source: §5.2 · p. 6
• Users said situated inspection clarified when and why models failed and could support reliability checks before high-stakes use, though trust impressions were mixed.One participant reported trusting the model less after discovering failures.Source: §5.2 · p. 6
• Compared with static desktop inspection, participants found AR more engaging and useful for viewpoints, while noting that screens better support statistical tests.An industrial participant preferred the system for testing from different viewpoints.Source: §5.2 · p. 6
• Most participants opposed adding LLM explanations because of visual clutter and reliability concerns, while two saw potential for optional support or less-experienced users.The authors therefore did not add an LLM feature.Source: §5.2 · p. 6, §5.3 · p. 7

5.3 Discussion and iterative improvements

• Participants flagged confusing controls and headset weight as usability concerns, prompting a simpler interface with clearer separation of live, offline, and recording functions.The study also added a condition for inspecting pre-recorded test images.Source: §5.2 · p. 6, §5.3 · p. 7

6.1 Participants and study design

• Sixteen participants with AI/ML experience completed a randomized, counterbalanced within-subject comparison of AR explanations and heatmaps on pre-recorded 2D images.Both conditions used the AR headset, and participants completed questionnaires and an interview.Source: §6.1 · p. 7

6.2 Quantitative results

• 3D and 2D had comparable usability and pragmatic quality, but 3D scored higher on hedonic quality; trust did not differ significantly.The 3D interface had significantly higher hedonic quality (p < .001), while the SUS difference was not significant (p = .312).Source: §6.2 · pp. 7–8
• Greater prior AR familiarity correlated with higher AR usability and pragmatic quality and lower workload, while AI/ML and explainability familiarity did not correlate significantly with trust.The reported correlations were ρ = 0.69 for usability, ρ = 0.67 for pragmatic quality, and ρ = −0.66 for workload.Source: §6.2 · p. 8
• 13 of 16 participants (81.3%) preferred 3D AR over 2D, despite significantly higher overall workload in 3D.The 2D condition had a mean workload of 6.19 (SD = 1.70), versus 8.02 (SD = 2.25) for 3D.Source: §6.2 · pp. 7–8

6.3 Qualitative results

• Participants valued moving around objects and viewing spatially registered explanations, describing AR as engaging and useful for seeing model behavior in real-world context.They also valued live interaction with the real environment and continuity over time.Source: §6.3 · p. 8
• Participants said live views and recording supported rapid, controllable data capture, while heatmaps helped reveal model features, confusing cases, and failure points.Users also described the approach as useful for debugging.Source: §6.3 · p. 8
• Interviewees described mixed effects of model understanding on trust and proposed applications including industrial inspection, medicine, and food sorting.Some said insight increased trust; others said discovering failures reduced it or did not change their trust.Source: §6.3 · p. 9

6.4 Discussion of results

• The discussion links AR’s higher workload to its 3D interaction demands, while noting users’ positive engagement and preference for its live, multi-view model inspection.Participants with more AR experience also rated the interface as more usable and reported lower workload.Source: §6.4 · p. 9

7.1 Lessons learned from the user evaluations

• Users described movement, interactivity, and direct coupling to the 3D world as central to understanding and debugging models.The majority preferred the AR tool for model inspection.Source: §7.1 · p. 9
• Participants accepted greater mental effort for an engaging, rewarding AR experience, contrasting desktop efficiency with AR engagement and exploration.Source: §7.1 · p. 9
• Users envisioned situated explainability for real-world fault inspection, data-bias discovery, decision-making, and informed data capture across industrial, medical, and home applications.Source: §7.1 · p. 9

7.2 Limitations and future work

• The studies involved a limited number of participants with AI/ML backgrounds, leaving novice understanding and needed interpretive support open for future study.Source: §7.2 · p. 9
• Offline mode supports only static objects because heatmaps are accumulated and objects are not tracked; the implementation is a proof of concept, with interface alternatives still to evaluate.Source: §7.2 · pp. 9–10

• AR explainability created a more engaging experience but higher cognitive workload than traditional explainability workflows.Source: §8 · p. 10
• The paper presents situated explainability as a starting point for using AR to support transparency, understanding, and debugging of AI solutions.Source: §8 · p. 10
• Situated explainability draws on AR’s spatial character for natural, real-time movement and interaction while testing and understanding AI in real environments.Source: §8 · p. 10