Through Human Eyes and Machine Eyes: Understanding View Mismatch in Video See-Through Extended Reality

1 Introduction

• Captured first-person views support downstream XR tasks and are often treated as proxies for the user's visual context.VST systems provide control over the rendered view, but the captured view does not necessarily represent what the user effectively sees.Source: §1 · p. 1
• The rectangular system frame can include peripheral or corner content outside the user's effective visible field.Figure 1 contrasts a Meta Quest 3 screenshot with an approximate human-visible view captured by a USB camera behind the headset lens; that view is illustrative, not an exact measurement of human vision.Source: Figure 1 · p. 2, §1 · p. 2
• The paper formalizes the mismatch and conducts a pilot-level boundary measurement on Meta Quest 3.The measurement provides a setup-specific estimate of the human-visible boundary.Source: §1 · p. 2
• The paper develops a threat and impact model for screenshot-based XR and tests representative cases spanning prompt injection, privacy leakage, and both directions of missing or unseen information.The cases illustrate potential consequences of system-visible but human-invisible content and human-visible but system-invisible content.Source: §1 · p. 2

2.1 Human Visual Fields and Visibility in Head-Mounted Displays

• Human visibility is asymmetric rather than rectangular, and HMD optics, eye position, eye relief, interpupillary distance, and fit shape what users can see.Prior XR work has mainly focused on perceptual quality and interaction performance, rather than whether captured views represent the user's effective visible region.Source: §2.1 · p. 3

2.2 System-Captured Views in Video See-Through XR Analysis

• Automated XR analysis has used system-captured first-person views, and recent systems increasingly use them as inputs to VLM perception and reasoning.This use can assume that captured views represent the region effectively visible to the user.Source: §2.2 · p. 3

2.3 Safety, Privacy, and Reliability Risks in AI-Integrated XR

• Prior AI-XR work examines prompt injection [19], privacy preprocessing and sensitive-content detection, and misleading or obstructive AR, generally without checking alignment with the human-visible region.The paper considers both system-visible but human-invisible content that can influence downstream models and human-visible but system-invisible content that can make screenshot-based analysis incomplete.Source: §2.3 · pp. 3–4

3 View Mismatch: Definition and Measurement

• The section frames the measurement as an initial empirical probe, not an exhaustive characterization of user- and device-dependent factors.Source: §3 · p. 4, §3.2 · p. 5

3.1 Problem Definition

• The model separates system-captured S and human-visible H into co-visible Rc, system-only Rs, and human-only Rh regions; cropping alone does not ensure alignment.Source: §3.1 · p. 4, Figure 2 · p. 5

3.2 Human-View Boundary Measurement Setup

• The single-observer Quest 3 pilot manually sampled a grid under fixed binocular viewing without head restraint, yielding only a setup-specific approximate boundary.Source: §3.2 · pp. 5–6

3.3 Measurement Results and Analysis

• Quest 3 showed a rectangular screenshot versus an approximately elliptical visible boundary, with a wider left system-only region likely related to left-eye capture.Source: Figure 4 · p. 7, §3.3 · p. 7

• The model separates four effects—prompt injection, privacy leakage, human-invisible information bias, and missing human-visible information—with the first three arising from Rs and the last from Rh.The figure distinguishes system-visible content outside the human-visible region from human-visible content absent from the screenshot.Source: §4 · p. 8, Figure 5 · p. 9
• System-only visual prompts may reach a downstream model without the user noticing, while sensitive information in Rs may enter screenshots, logs, remote streams, or model inputs beyond the user's perceived exposure.Virtual prompts can remain positioned outside the effective visible field while inside the captured frame; this creates a gap between perceived and actual privacy exposure.Source: §4 · p. 8
• Task-relevant objects in Rs can bias scene understanding, producing incorrect task-state estimates, misleading assistance, or inaccurate attention inference.The user may not effectively perceive information incorporated by the downstream model.Source: §4 · p. 8
• Content in Rh may be absent from screenshots or model inputs, leaving screenshot-based records and post-hoc analysis incomplete or misleading about what the user saw.Source: §4 · p. 8

5.1 Demonstration Setup

• The demonstrations used four everyday scenarios, Meta Quest 3 screenshots overlaid with an estimated human-visible boundary, and task-specific queries to GPT-5.4.A Unity-based Quest 3 application placed virtual elements inside or outside the estimated boundary; the cases are illustrative demonstrations rather than an exhaustive benchmark or statistical evaluation.Source: §5.1 · p. 9, §5.2 · p. 10

5.2 Case Analysis

• A prompt-injection message in Rs shifted GPT-5.4 from identifying the stop sign to giving a vague scene-level answer, despite the user being instructed to verify the prompt remained outside the effective visible field.The answer was “The road intersection,” rather than a concrete safety-related object.Source: §5.2 · p. 10
• A fabricated credit card placed in Rs was identified by GPT-5.4, which extracted its card number despite the information remaining outside the observer's effective visible field.Source: §5.2 · p. 10
• A human-invisible stop sign biased GPT-5.4's safety-hazard answer away from the knife, while human-visible contact information omitted from the screenshot left the model unable to answer how to reach the sales representative.The stop sign's usual safety meaning was weakened in the indoor scene; the missing contact information illustrates the opposite mismatch direction.Source: §5.2 · p. 11
• Across four demonstrations, both Rs content unavailable to the user and Rh content absent from the screenshot affected downstream VLM tasks, producing security, privacy, and task-effectiveness failure modes.Source: §5.2 · p. 11

• XR platforms could configure screenshots or exported frames to better approximate the user's effective visible field, conservatively reducing Rs in security- and privacy-sensitive pipelines even if Rh grows.The authors distinguish potentially silent downstream effects or privacy exposure from Rs from record incompleteness due to Rh.Source: §6 · p. 11
• Downstream AI pipelines could use approximate visible-region masks to ignore, blur, downweight, or label content outside the estimated human-visible region.Masks could be estimated from device configuration, calibration results, or conservative geometric assumptions.Source: §6 · p. 11
• XR systems could provide a permission request, preview, or visible-region overlay before screenshots are stored, transmitted, or analyzed, making capture boundaries more transparent without eliminating mismatch.Source: §6 · p. 12
• The boundary measurement used one Meta Quest 3 observer and one setup without repeated measurements, while the risk analysis used representative case studies rather than a systematic benchmark and did not quantify mismatch frequency.The measurement also did not report area-based measures, and the static frame-based analysis did not characterize changes over natural head or gaze movements.Source: §6 · p. 12

• The Meta Quest 3 pilot measurement found a clear discrepancy between the rectangular screenshot frame and the approximate human-visible boundary.Source: §7 · p. 12
• The paper developed a risk and impact model and illustrated security, privacy, and reliability failure modes arising from view mismatch.Source: §7 · p. 12
• The findings suggest screenshot-based XR systems should account for differences between system-visible and human-visible content in downstream sensing and VLM analysis.Source: §7 · p. 12