Source-linked AI summary
Why User Studies and Participant Experience Reporting Matter for VR Motion Privacy?
Azim Ibragimov, Eric D. Ragan
TL;DR
Public VR motion recordings can expose users to identification and profiling, while privacy transformations may alter how their performance appears. This study compares physical deviation and user evaluations across three mechanisms and five deviation levels, finding that user evaluations explain more variation in acceptance and that prior experience matters.
Problem
The paper asks whether physical deviation explains acceptance of protected VR motion as well as user evaluations, against the backdrop of privacy risks from persistent motion recordings.
Source: §1 · pp. 1–2Method
A controlled study had 286 participants evaluate recordings protected by three mechanisms at five physical-deviation levels and examined the role of prior experience.
Source: §1 · p. 2Results
Both physical deviation and user-facing evaluations were significantly associated with acceptance, but user-facing evaluations explained substantially more variation, and prior experience significantly affected sharing acceptance.
Source: §1 · pp. 2–3Takeaways & Limitations
The authors recommend combining physical-deviation measures with user studies and collecting, reporting, and considering participants’ prior experience when interpreting results.
Source: §1 · p. 3Only 24% of participants reported prior privacy-mechanism experience, and participants with higher experience remained uncommon even in the university recruitment setting.
Source: §6.3 · p. 21
Abstract
from arXiv · showhide
Public VR game leaderboards contain tracked motion recordings uploaded by hundreds of thousands of users. Once uploaded, these recordings are accessible to anyone and create privacy risks (i.e., identification and profiling). Prior work has proposed mechanisms that modify tracked movement to reduce these risks. Their utility is commonly evaluated through physical deviation, where smaller deviations indicate better utility, while user studies are less common. However, it remains unclear how well physical deviation explains users' acceptance of a mechanism compared to user studies. We examine this through a user study of three VR motion privacy mechanisms at five physical deviation levels. We find that user studies explain substantially more variation in mechanism acceptance than physical deviation, although physical deviation remains significant. We also find that prior VR experience and exposure to VR privacy mechanisms significantly affect acceptance. We recommend combining physical deviation with user studies and reporting participants' prior experience.
1 Introduction
Public VR leaderboards preserve motion recordings for score verification, but those recordings can expose users to identification and profiling. This study compares physical deviation and user-facing evaluations as explanations of acceptance, while examining prior experience.
1 Introduction
- Public leaderboard recordings persist after upload and can expose users to identification and profiling through distinctive motion patterns and other sensitive information.One cited study re-identified 55,541 Beat Saber users from uploaded recordings.Source: §1 · p. 1
- Privacy mechanisms modify motion to reduce identifiability, but changes such as jitter, smoothing, or discontinuities can make protected recordings misrepresent expected performance.Such changes may make users less willing to share recordings they consider inaccurate, delayed, or otherwise unlike their performance.Source: §1 · pp. 1–2
- Utility is commonly assessed through physical deviation, where smaller deviations imply better utility, while user studies directly assess how people perceive protected motion.Prior findings suggest usability effects can differ across mechanisms as their strength increases.Source: §1 · p. 2
- The controlled study evaluated three mechanisms at five physical-deviation levels with 286 participants viewing Beat Saber leaderboard recordings and rating degradation, privacy risk, trust, and acceptance.The five deviation levels were 1, 3, 5, 7, and 11 cm.Source: §1 · p. 2
- Both physical deviation and user-facing evaluations are significantly associated with acceptance, but user-facing evaluations explain substantially more variation than physical deviation alone.The authors recommend combining physical-deviation measures with user studies rather than relying on either alone.Source: §1 · pp. 2–3
- Participants’ prior experience significantly affected sharing acceptance, leading the authors to recommend collecting and reporting experience with VR and VR privacy mechanisms.They also recommend considering this experience when interpreting study results.Source: §1 · p. 3
2 Background and Related Work
The background connects risks in public VR motion data with privacy mechanisms and their utility evaluations. It motivates comparing user acceptance with physical-deviation measures and accounting for participant experience.
2.1 Privacy Risks of VR Motion Data
- VR motion can support re-identification, account linking, and inference of demographic, health, cognitive, or behavioral information from publicly accessible recordings.Prior work linked over 4 million recordings from more than 55,000 users with 94% accuracy.Source: §2.1 · p. 3
2.2.1 Types of VR Privacy Mechanisms.
- Access controls, partial releases, and cryptographic approaches can conflict with public access or score verification, motivating perturbation mechanisms that retain shareable recordings but alter motion.Gaussian noise adds jitter, smoothing changes trajectories, and temporal downsampling reduces unique timestamps; these changes can impair perceived motion accuracy or performance representation.Source: §2.2.1 · p. 4
2.2.2 Utility Evaluations of VR Privacy Mechanisms.
- Utility evaluations use distance-based distortion measures or user studies, but greater physical deviation does not affect usability uniformly across mechanisms.Wilson et al. [44] found increasing mechanism strength affected Gaussian-noise usability more substantially than smoothing; Table 1 reports nine distance-based studies, three user-study studies, and one using both.Source: §2.2.2 · p. 5, Table 1 · p. 5
- Prior work rarely evaluated the same mechanisms with both approaches, leaving the relationship between physical deviation and acceptance unclear; this work compares them and examines prior experience.The authors also examine whether users’ prior experience affects their evaluation of different mechanisms.Source: §2.2.2 · pp. 5–6
3 Threat Model
The threat model considers a public VR leaderboard that verifies scores before publishing privatized motion recordings. It specifies the protected asset, an adversary acting on public recordings, and post-hoc privacy application by the leaderboard.
3 Threat Model
- The setting is a VR leaderboard where users submit motion recordings for score verification before a trusted service privatizes and publishes them.Recordings may include head pose, controller trajectories, timing, and other movement-derived signals.Source: §3 · p. 6, §3.1 · p. 6
3.1 System Se!ing and Protected Asset
- The protected asset is the original gameplay recording, which contains movement information needed to reconstruct and verify gameplay.The leaderboard applies protection after verification and before public release.Source: §3.1 · p. 6, §3 · p. 6
3.2 Adversary Model
- The adversary can download or scrape public privatized recordings and use motion trajectories for identification, linkage, or profiling without access to the headset, network, or original recording.Goals may include linking recordings across different usernames or avatars and inferring behavioral, physical, or health-related characteristics.Source: §3.2 · pp. 6–7
3.3 Privacy Mechanism and Scope
- Privacy is applied post-hoc by the trusted leaderboard after score verification, rather than by users during gameplay.The paper excludes real-time user-applied mechanisms because modifications could affect gameplay-relevant motion and potentially provide an unfair advantage.Source: §3.3 · p. 7
4 Method
The study evaluated post-hoc VR motion privacy mechanisms using Beat Saber replay recordings, varying mechanism type and physical deviation while measuring user judgments. Its design also collected prior experience to examine how it related to acceptance.
4.1 Research "estions
- The study asked whether physical deviation or user-facing perceptions better explain acceptance, and whether prior VR and privacy-mechanism experience shape acceptance.These questions covered differences across mechanism types and deviation levels.Source: §4.1 · p. 7
4.2 Study Design Overview
- Participants evaluated Beat Saber leaderboard replays under one assigned mechanism, across five within-participant deviation levels: 1, 3, 5, 7, and 11 cm.The mechanism assignment was between participants; the deviation levels were within participants.Source: §4.2 · pp. 7–8, Figure 2 · p. 8
4.3 Privacy Mechanisms
- Gaussian noise, smoothing, and temporal downsampling were selected because prior work evaluated them with both user studies and distance metrics; they alter motion in different ways.Gaussian adds jitter, smoothing attenuates rapid movement, and temporal downsampling produces stair-step motion.Source: §4.3 · pp. 8–9, Figure 3 · p. 9
4.4 Measures
- Five constructs—spatial and temporal precision, privacy risk, trust, and acceptance—used three 7-point Likert items averaged per construct, with acceptance as the primary outcome.The measures capture user judgments that physical-deviation metrics cannot directly observe.Source: §4.4 · pp. 9–11, Figure 5 · p. 11
4.5 Procedure
- Participants reported prior experience before instruction and a comprehension check, then rated five deviations for one randomly assigned, unnamed mechanism; practice ratings were excluded.Participants could proceed to the rating task only after passing the comprehension check within three attempts.Source: §4.5 · pp. 12–13
5.1 Participants
The analysis used 286 retained participants recruited from university computer science courses, with prior VR experience common and prior privacy-mechanism experience less common.
5.1 Participants
- Of 364 respondents, 286 remained after quality control; 78% reported prior VR experience, while 24% had experience with VR privacy mechanisms.Participants were mostly male (65%), and ages ranged from 18 to 47 years, with a median of 20.Source: §5.1 · p. 13
5.2 "ality-control criteria
The study excluded incomplete responses, straightlining, and completion times outside the 8-to-60-minute range.
5.2 "ality-control criteria
- Responses were excluded for incompleteness, straightlining, or completion times below 8 minutes or above 60 minutes.The authors set the thresholds using their timing estimates of the study flow and allowed flexibility for slower completion or minor interruptions.Source: §5.2 · p. 13
5.3 Analysis #1: Explaining User Acceptance
The analysis compares physical deviation and user-facing perceptions as predictors of acceptance. User-facing ratings explain substantially more variation, while physical deviation adds a small but significant contribution.
5.3.1 Method of Analysis #1.
- Respondent-demeaned regression models compare physical deviation, user-facing ratings, and their combination in explaining within-participant acceptance changes.The ratings include perceived spatial and temporal degradation, perceived privacy risk, and trust.Source: §5.3.1 · p. 14
5.3.2 Results of Analysis #1.
- User-facing ratings explain substantially more acceptance variation than physical deviation alone, while adding deviation to ratings yields only a small increase.The combined model explains the most variation; physical deviation remains significantly associated with acceptance.Source: §5.3.2 · p. 14, Figure 8 · p. 15
- In the combined model, trust is positively associated with acceptance, while perceived degradation and privacy risk are negatively associated.Trust has the strongest association among these ratings; physical deviation has a comparatively small coefficient.Source: §5.3.2 · pp. 14–15
- Physical deviation remains significantly associated with acceptance after accounting for user-facing ratings, so the two provide complementary information.Its association is comparatively small, and physical deviation alone explains little within-participant variation.Source: §5.3.2 · pp. 14–16
5.4 Analysis #2: Participant Experience and Sharing Acceptance
The analysis tests whether prior VR and VR privacy-mechanism experience relate to acceptance across mechanisms and deviation levels. Both experience types are significantly associated with acceptance, with differences varying across conditions.
5.4.1 Method of Analysis #2.
- A linear mixed-effects model accounts for repeated participant ratings while testing experience, mechanism, deviation level, and their two-way interactions.Experience was grouped into no, novice, and intermediate/expert levels; Holm-adjusted contrasts characterized significant effects.Source: §5.4.1 · p. 16
5.4.2 Results of Analysis #2.
- Prior VR experience and VR privacy experience have significant main effects, and four two-way interactions are significant; mechanism type and deviation level have no significant main effects.The VR privacy experience-by-mechanism interaction is not significant.Source: §5.4.2 · p. 16, §5.4 · p. 16, Figure 9 · p. 17
- Novice VR users report higher overall acceptance than intermediate/expert users, while intermediate/expert VR privacy users report higher acceptance than no-experience and novice groups.These are significant Holm-adjusted pairwise contrasts.Source: §5.4.2 · p. 17, Figure 10 · p. 18
- For Gaussian and Temporal mechanisms, acceptance differs by VR experience; no significant experience differences appear for Smoothing.Novice users report higher acceptance for Gaussian than no-experience and intermediate/expert users; for Temporal, no-experience and novice users report higher acceptance than intermediate/expert users.Source: §5.4.2 · p. 17, Figure 11 · p. 18
- Experience-related acceptance differences also vary across physical deviation levels, including higher acceptance among novice users than intermediate/expert users at High and Very High levels.At Very Low deviation, novice VR users report higher acceptance than users with no VR experience.Source: §5.4.2 · pp. 17–18, Figure 12 · p. 19
- At Normal, High, and Very High deviation levels, intermediate/expert VR privacy users report higher acceptance than less-experienced privacy-mechanism groups in the stated pairwise contrasts.At Normal, this group reports higher acceptance than both novice and no-experience participants; at High, higher than both groups.Source: §5.4.2 · pp. 18–19
6 Discussion
The discussion recommends evaluating protected motion with both physical deviation and user-facing measures, while accounting for participants’ prior experience. It also outlines replay-sharing design implications and boundaries for interpreting the findings.
6.1 Key Takeaways
- User-facing ratings explained substantially more variation in sharing acceptance than physical deviation alone, although physical deviation remained significantly associated with acceptance.The authors recommend using both measures rather than replacing physical deviation with user-facing evaluations.Source: §6.1 · pp. 19–20
- Prior VR and privacy-mechanism experience were associated with acceptance, with different patterns for general VR experience and mechanism experience.Novice VR users reported higher acceptance in several comparisons, while greater privacy-mechanism experience was associated with higher acceptance, particularly at moderate-to-high deviation levels.Source: §6.1 · p. 20
6.2 Design Implications for VR Replay-Sharing and Leaderboard Platforms
- Leaderboard platforms could let users preview protected replays and choose which version to publish.The proposed workflow generates versions using different mechanisms or protection levels after verifying the original replay.Source: §6.2 · pp. 20–21
- Platforms could present choices through motion properties users can evaluate, while retaining physical deviation as an internal technical measure.Examples include preserving movement appearance, timing, or smoothness, or prioritizing stronger protection despite more noticeable changes.Source: §6.2 · pp. 20–21
6.3 Limitations and Future Work
- Designing mechanisms around users’ perceptions may allow greater physical deviation and stronger privacy protection while keeping motion acceptable.The authors identify this as a possibility for future work, not an established outcome.Source: §6.3 · p. 21
- Experienced participants were uncommon: even in the university-recruited population, only 24% reported prior VR privacy-mechanism experience.The authors suggest longitudinal studies that repeatedly expose participants to mechanisms to examine how evaluations change with familiarity.Source: §6.3 · p. 21
- The study focused on leaderboard threats; multiplayer, social, and casual VR may involve different privacy risks and requirements.Real-time mechanisms were unsuitable for this leaderboard setting because original motion was needed to verify performance, but may better fit active VR interaction.Source: §6.3 · p. 21
7 Conclusion
A 286-participant study found that user-facing perceptions explained more variation in sharing acceptance than physical deviation, while both and prior experience mattered. The paper recommends combining evaluation measures and reporting participant experience.
7 Conclusion
- In a controlled study of 286 participants, user-facing perceptions explained substantially more variation in acceptance than physical deviation alone, while both were significantly associated with sharing acceptance.The paper also reports that participants’ prior experience significantly affected acceptance and recommends collecting and reporting that experience.Source: §7 · p. 22
8 Open Science
The paper releases study materials as supplementary material.
8 Open Science
- The supplementary material includes the instructional module and video stimuli presented to study participants.The materials supported the VR motion privacy introduction and privacy-mechanism rating task.Source: §8 · p. 22