Source-linked AI summary
Hidden In Plain Gaze: Gaze Representations as Privacy Controls for Utility and Re-identification Risk in XR
Cory Ilo, Brendan-David John, Doug A. Bowman
TL;DR
XR gaze and head signals support intent and task inference but can also reveal biometric identity. This paper compares gaze representations under matched capacity to test representation choice as a privacy control. Engineered features preserve most action-recognition utility while substantially reducing, but not eliminating, re-identification.
Problem
XR systems need gaze for interaction and inference, but the same signals encode persistent biometric identity, creating a privacy–utility challenge.
Method
Using HoloAssist, the study compares raw gaze, spatial heatmaps, and engineered eye-movement features under matched model capacity for action recognition and closed-set re-identification.
Results
Engineered features retain about 85% of raw gaze’s action-recognition accuracy while reducing re-identification from ≈39× chance to ≈3.9× chance across N=206 identities.
Takeaways & Limitations
Representation choice is a practical privacy lever: engineered features provide an interpretable and auditable utility–privacy operating point, although abstraction alone does not guarantee privacy.
Takeaways & Limitations
The findings rely on the single HoloAssist corpus, whose long-tailed distribution interacts with representation and leaves cross-dataset replication open.
Abstract
from arXiv · showhide
Intelligent extended reality (XR) systems increasingly use eye and head tracking to infer user intent, task, and attention, but the same signals can also reveal biometric identity. We study whether gaze data representation choice can serve as a lightweight privacy control at feature extraction, before adding perturbation or formal privacy mechanisms. Using the egocentric HoloAssist dataset, we compare three gaze representations under matched model capacity: raw gaze, spatial attention heatmaps, and engineered eye-movement features. We evaluate each representation on action recognition as task utility and closed-set user re-identification as privacy leakage. Representation choice substantially changes the privacy-utility tradeoff. Engineered features retain roughly 85% of raw gaze's action-recognition accuracy while reducing re-identification by about an order of magnitude, to roughly four times the chance rate across 206 identities. This reduction attenuates rather than eliminates identity leakage, and the differences across representations show that abstraction alone does not guarantee privacy. Engineered features expose interpretable and auditable structure, giving designers a transparent privacy lever that complements mechanisms such as differential privacy.
1 INTRODUCTION
XR gaze and head signals support context-aware interaction but also encode biometric identity, creating a privacy–utility challenge. This study tests whether representation choice can reduce identity leakage while preserving task utility.
- Gaze and head motion support inference of user intent, task state, and attentional focus while also revealing distinctive biometric patterns.
- Identity-related information is associated with fine-grained temporal dynamics, whereas task-relevant information often appears in coarser spatial attention distributions.
- Existing privacy mechanisms can reduce utility or add latency, motivating representation-level intervention before perturbation or formal privacy mechanisms.
- The study compares raw gaze, engineered eye-movement features, and spatial heatmaps on task classification utility and user re-identification risk using HoloAssist.
- Representation choice acts as a practical privacy lever, with engineered features offering interpretable and auditable structure rather than eliminating identity leakage.
- Engineered features retain about 85% of raw gaze’s action-level top-1 accuracy while reducing re-identification from ≈39× chance to ≈3.9× chance across N=206 identities.
2 RELATED WORK
XR telemetry supports adaptive interaction but functions as a behavioral biometric record, creating linkage and privacy risks. Existing protections provide partial remedies, while representation-level abstraction remains an empirical question rather than an assumption of privacy.
- Gaze provides continuous information about attention, task state, and intent, while task-dependent eye–head dynamics can also distinguish users.
- XR head, hand, and gaze streams can identify individuals even without direct names or account identifiers.
- Linkage across contexts is the central privacy risk because biometric signatures can connect telemetry to external profiles and sensitive attributes.
- Differential Privacy and Federated Learning offer formal or architectural protections but face repeated-release, unit-of-privacy, utility, latency, and update-leakage challenges in XR.
- Anonymization and real-time perturbation remain application-dependent because removing identifiers or altering temporal precision may not prevent re-linking while preserving utility.
3 THREAT MODEL
The threat model examines whether derived gaze representations still permit user re-identification and thereby enable cross-context linkage. It measures identity leakage specifically, while excluding attribute inference and several orthogonal attack surfaces.
- The central question is whether a third party can recognize the same person from gaze and whether representation choice changes that difficulty.
- The trusted XR layer computes Z=f(X), exposing only the derived representation to third parties while retaining raw stream X.
- The protected asset is any persistent biometric signature remaining in the derived representation, because leaked biometric signatures cannot be reissued.
- The adversary is an honest-but-curious recipient that legitimately accesses Z, knows f, and trains a re-identification model for linkage across sessions, applications, or datasets.
- Identity leakage is measured with closed-set Top-1 and Top-k identification against the 1/N chance baseline, rather than one-to-one authentication.
- The study measures identity leakage, not attribute inference, and excludes system compromise, physical side channels, traffic analysis, social engineering, and multimodal linkage.
4 METHODOLOGY
The methodology compares three gaze representations derived from synchronized eye and head streams on HoloAssist, using matched encoders and training to evaluate utility and privacy. The representations range from minimally processed temporal signals to abstract spatial and engineered summaries.
- The study compares raw gaze, spatial attention heatmaps, and engineered features on coarse-grained action recognition and closed-set user re-identification.
- HoloAssist provides synchronized eye-gaze and head-pose streams from naturalistic physical manipulation tasks performed with mixed-reality headsets.
- The released data contains 12,542 action events across 1,758 sessions and more than 200 identities; the experiment filters this to 10,212 events, 1,754 sessions, and 85 action classes.
- Representations: Raw gaze preserves eye and head movements as multichannel time series, including Eye-in-Head, head rotational velocity, and Gaze-in-World streams.
- Representations: Heatmaps combine gaze and head pose into a global look direction, project it onto a 360° by 180° grid, and accumulate points into image-like spatial summaries.
- Representations: Engineered features summarize each sliding analysis window with a 55-dimensional vector describing gaze velocity, acceleration, position, and dispersion.
- Representations: The representations form a spectrum from raw temporal signals to heatmaps and compact summary statistics, with abstraction discarding increasingly fine-grained motion structure.
- Model comparison: All branches use a common CNN-plus-BiGRU backbone, with input-shape-driven architectural differences and matched capacity, normalization, classifier depth, and training budget.
5 EXPERIMENTAL RESULTS
Across matched evaluation settings, engineered gaze features preserve substantially more action-recognition utility than heatmaps while reducing re-identification much more strongly than either alternative. The privacy reduction remains above chance, so representation choice attenuates rather than eliminates identity leakage.
- Evaluation setup: All 206 identities appear in train, validation, and test, with CGA segments—not whole sessions—held out under an identity-stratified 80/10/10 split.This measures within-identity, cross-segment performance and separates representation effects from identity coverage.
- Utility: Action Classification: Raw gaze ranked highest for action recognition, engineered features followed closely, and heatmaps performed worst across top-1, top-5, and balanced accuracy.The same ordering held across the reported action-recognition metrics.
- Utility: Action Classification: 0.033 top-1 accuracy was the engineered representation’s mean deficit versus raw gaze on the primary action task, a roughly 15% relative reduction from raw’s 0.216.The 90% confidence interval was [0.014, 0.052], and the pre-registered equivalence test was inconclusive at Δ = 0.05.
- Utility: Action Classification: 0.113 top-1 accuracy was the heatmap’s mean deficit versus raw gaze, with substantially greater across-seed variability and no seed near the Δ = 0.05 equivalence bound.The engineered representation therefore incurred a smaller and more stable utility cost than the heatmap.
- Privacy: Re-Identification: 10.1× was the engineered representation’s reduction in top-1 re-identification relative to raw gaze, from 0.1917 to 0.0190.Its top-1 value remained about 3.9× the 1/206 chance rate, while heatmap top-1 re-identification fell only 1.47×.
- Overall trade-off: About 85% of raw gaze’s action-recognition utility was retained while re-identification fell by roughly an order of magnitude, but the remaining above-chance signal shows leakage was attenuated rather than eliminated.The comparison spans the utility and privacy axes under the study’s representation-level evaluation.
6 DISCUSSION, LIMITATIONS, & FUTURE WORK
The engineered gaze representation offers the strongest observed privacy–utility tradeoff, retaining most task signal while substantially reducing identity leakage relative to raw gaze. Interpretation is bounded by representation-specific architectural constraints, an uncalibrated equivalence bound, and reliance on HoloAssist.
- Findings: Roughly 85–90% of raw gaze’s task signal is retained while closed-set re-identification falls by roughly an order of magnitude.The engineered representation provides the most favorable privacy–utility position observed, but does not eliminate above-chance identity leakage.
- Findings: The engineered representation trades a modest utility cost for a large privacy gain, placing it in a design region raw gaze cannot reach.This exchange reflects most utility retained alongside substantially less identity leakage.
- Design implication: The fixed deterministic transform acts at feature extraction without privacy budget, calibrated noise, or inference-time perturbation.It is therefore a lightweight complement to data-level mechanisms rather than a replacement for them.
- Limitations: The heatmap comparison is constrained because all branches use a 1D-oriented backbone structurally mismatched to the heatmap’s 2D grid.A representation-tailored architecture search could change the observed heatmap result.
- Limitations: The findings are bounded by a matched-budget regime, an equivalence bound set a priori, and evaluation on a single corpus.HoloAssist’s long-tailed distribution interacts with representation in ways single-dataset evaluation cannot disentangle.
- Future work: A representation-appropriate backbone study and joint privacy–utility optimization are proposed to test whether better operating points are achievable.The paper also identifies larger, more ecologically diverse gaze corpora and runtime-exposure evaluation as future priorities.
7 CONCLUSION
The paper positions gaze representation as a privacy lever applied at feature extraction. A continuous engineered feature set retains most action-classification utility while reducing re-identification to roughly four times chance, and its inspectable inductive biases complement differential privacy.
- A continuous engineered feature set retains most action-classification utility while reducing re-identification to roughly four times chance across 206 identities.
- Hand-designed features provide inspectable inductive biases that let designers reason about what the encoding can reveal about identity.This transparency complements rather than replaces data-level mechanisms such as differential privacy.