PaperlayerView exact revision on arXiv

Source-linked AI summary

The Dark Side of Perceptual Manipulations in Virtual Reality

Wen-Jie Tseng, Elise Bonnail, Mark McGill, Mohamed Khamis, Eric Lecolinet, Samuel Huron, Jan Gugenheimer

arXiv:2202.13200v1cs.HC

TL;DR

VPPMs can nudge users’ physical movements in VR, but imperceptibility and ambiguity around consent, awareness, and intent create potential for harm. The paper explores these risks through speculative workshops and sample applications, then proposes mitigations and recommendations.

  • Problem

    VPPMs can alter physical movement below perception thresholds, while ambiguity around consent, awareness, and intent creates potential for harm and a research blind spot.

    Source: §1 · p. 1, §8 · p. 11

  • Method

    The paper uses speculative design workshops to classify potential attacks and characterize harm, implements two sample applications, and proposes mitigations and recommendations.

    Source: §1 · p. 2

  • Results

    The paper identifies puppetry and mismatching attacks, characterizes potential physical harm, and demonstrates trivial subversion through two sample applications.

    Source: §9 · p. 12

  • Takeaways & Limitations

    The paper proposes platform-level mitigations and recommendations for practitioners and researchers against malicious VPPM use.

    Source: §9 · p. 12

    The workshop method cannot assess the likelihood of malicious VPPM attacks or the occurrence of physical harm in everyday VR use.

    Source: §8.1 · p. 11

Abstract

from arXiv · show

"Virtual-Physical Perceptual Manipulations" (VPPMs) such as redirected walking and haptics expand the user's capacity to interact with Virtual Reality (VR) beyond what would ordinarily physically be possible. VPPMs leverage knowledge of the limits of human perception to effect changes in the user's physical movements, becoming able to (perceptibly and imperceptibly) nudge their physical actions to enhance interactivity in VR. We explore the risks posed by the malicious use of VPPMs. First, we define, conceptualize and demonstrate the existence of VPPMs. Next, using speculative design workshops, we explore and characterize the threats/risks posed, proposing mitigations and preventative recommendations against the malicious use of VPPMs. Finally, we implement two sample applications to demonstrate how existing VPPMs could be trivially subverted to create the potential for physical harm. This paper aims to raise awareness that the current way we apply and publish VPPMs can lead to malicious exploits of our perceptual vulnerabilities.

1 INTRODUCTION

VPPMs use perceptual thresholds to nudge physical movement for VR interaction, but ambiguous consent, awareness, and intent create potential for abuse and harm. The paper defines the threat, explores it through workshops and sample applications, and proposes mitigations.

1 INTRODUCTION

  • VPPMs are XR-driven exploits that alter multisensory perception of physical actions and reactions to nudge users’ movements.They exploit perceptual thresholds to overcome VR limitations and enable new interactions.Source: §1 · p. 1
  • Ambiguous consent, awareness, knowledge, and intent can make VPPMs harmful, while published perception thresholds may be abused by malicious third parties.The authors identify a research blind spot where VPPMs are published without due consideration of potential harm.Source: §1 · pp. 1–2
  • The paper focuses on imperceptible, non-consensual manipulation of a user’s physical actions toward bodily harm, excluding devices that physically direct or override movement.The paper defines physical harm as hurt or damage relating to the VR user’s body.Source: §1 · p. 2
  • Two speculative design workshops classify potential attacks and characterize harm; two sample applications demonstrate how published VPPMs can be subverted, alongside proposed mitigations and recommendations.The three stated contributions are a definition, attack classification and harm characterization; two harmful applications; and recommendations for practitioners and researchers.Source: §1 · p. 2

2 THREAT MODEL

The threat model assumes an attacker compromises a VR system and uses a malicious application containing disguised VPPMs to manipulate a user imperceptibly.

2 THREAT MODEL

  • An attacker seeks to harm the VR user by compromising the VR system, for example by tricking the user into installing malware or a malicious app.The attacker may access environmental information through headset cameras, real-time movement data from headset and controller sensors, and standard VR application APIs.Source: §2 · p. 2
  • A malicious app can disguise VPPMs as part of its setup and imperceptibly manipulate actions such as walking or reaching.The attacker can use information about the user’s motion and safety boundaries, such as Oculus Guardian.Source: §2 · p. 2
  • Potential harms include tripping, hitting a wall, holding something dangerous, or walking into a dangerous area, with consequences that may include death.The passage situates these examples in accidents and bystander abuse.Source: §2 · p. 2

3 RELATED WORK

Related work connects VR presence and perceptual manipulation with ethical and security concerns, including the possibility that techniques designed for interaction could be misused.

3.1 VR Technologies and Experiences

  • VR presence, plausibility, and embodiment illusions support realistic interaction, and their effectiveness helps make VPPMs applicable across scenarios.Presence is the subjective feeling of being in a virtual environment.Source: §3.1 · pp. 2–3

3.2 Perceptual Manipulations in VR

  • VPPMs manipulate virtual-to-physical mappings, often below perception thresholds, to enable redirected walking, redirected touching and haptics, and changes to posture or fatigue.The related work also notes that VPPMs with positive intents may be exploited maliciously to provoke harm.Source: §3.2 · p. 3

3.3 The Potential Harm and Attacks in VR

  • VR’s immersive and persuasive effects can prompt physical actions that lead to harm, including collisions, falls, or sitting on a virtual chair without a real counterpart.The passage connects large-scale embodied movement in VR with greater likelihood of action-related physical harm than desktop or mobile interaction.Source: §3.3 · p. 3
  • Prior work demonstrated a Human Joystick Attack that directs an immersed user without their knowledge; this paper examines the broader class of attacks using VPPMs rather than technical vulnerabilities.The passage frames HCI as a meeting point for computer science, psychology, cognitive science, and design expertise relevant to human vulnerabilities.Source: §3.3 · p. 3

4 METHOD: SPECULATIVE DESIGN WORKSHOP

The authors use speculative design workshops to explore potential physical harm from malicious VPPMs, with participants generating scenarios, dimensions, and relevance ratings.

4 METHOD: SPECULATIVE DESIGN WORKSHOP

  • The workshop asked researchers and designers to brainstorm harmful VPPM scenarios, contextualize them with dimensions, and rate each dimension’s relevance to studying and preventing harm.The speculative design approach was chosen to critique current practices and reflect on future technologies and ethical implications.Source: §4 · p. 3

4.1 Participants

  • Eight VR/XR researchers and designers took part in two workshops of four, bringing varied research and design perspectives and at least average VR experience.Participants’ reported VR expertise averaged 3.75 on a five-point scale (SD=1.03).Source: §4.1 · pp. 3–4

4.2 Procedure

  • The four workshop steps were instruction, brainstorming, synthesizing, and voting; the instruction introduced VPPMs and framed speculation about abusive movement manipulation.The brainstorming premise assumed future VR with full-body tracking and imperceptible whole-body VPPMs.Source: §4.2 · p. 4, Figure 1 · p. 4
  • Participants individually brainstormed scenarios that manipulate body motions to provoke harm while preserving perceived agency, then presented and discussed their ideas.The task excluded motions created by external devices such as EMS or exoskeletons.Source: §4.2 · p. 4
  • Participants synthesized dimensions for positioning all scenarios, then rated each dimension’s relevance on a five-point scale within their own workshop.Brainstorming and synthesizing involved group discussion and collaboration; these steps were recorded.Source: §4.2 · pp. 4–5

5 WORKSHOP RESULTS

The workshops generated 19 scenarios, which the authors analyzed thematically to identify techniques and potential harms. The resulting scenarios include redirected actions, broken system trust, and obscured physical surroundings.

5.1 Data Analysis

  • Three authors jointly coded transcripts and scenarios using thematic analysis, resolving coding conflicts through discussion across nine sessions.Participants did not take part in the analysis; each coding session averaged two hours.Source: §5.1 · p. 5

5.2 Scenarios

  • The 19 scenarios describe techniques and potential harms, including redirection, disrupted system trust, and virtual content that occludes the physical world.Five scenarios were not directly associated with VPPMs; they included false information, advertising, an unspecified technique, and VR overuse.Source: §5.2 · p. 5

5.3 Classification of Attacks

The authors classify workshop scenarios as puppetry, mismatching, miscellaneous, or unclassified attacks. Puppetry redirects physical actions, while mismatching manipulates the correspondence between virtual objects and the physical environment.

5.3 Classification of Attacks

  • Open and axial coding identified puppetry and mismatching as the two main attack classes; scenarios without a broader theme were miscellaneous or unclassified.Two scenarios were unclassified because they were too specific and lacked technical detail.Source: §5.3 · p. 5

5.3.1 Puppetry Attacks.

  • Walking puppetry uses redirected walking to steer users toward locations that may cause harm, such as falling or entering a dangerous area.It appeared in five scenarios; arm-movement puppetry uses redirected haptics to guide a hand toward property or potentially harmful objects.Source: §5.3.1 · p. 5

5.3.2 Mismatching Attacks.

  • Mismatching attacks exploit differences between virtual objects and their physical counterparts to elicit user misinterpretation.The described environment is true-positive: each virtual object has a one-to-one real-world representation.Source: §5.3.2 · p. 5
  • False-positive attacks add a virtual object without a physical counterpart; habituation to a perfectly mapped environment and user trust can make interaction with it dangerous.Repeated tasks can build habituation until users act without conscious attention.Source: §5.3.2 · p. 5
  • False-negative attacks hide physical information, such as traffic noise or an object, leaving users unaware of dangers; hiding an open window could lead to a fall or fatal outcome.The passage notes that these attacks could occur in open spaces where the system must continually detect surroundings.Source: §5.3.2 · p. 6
  • Swapping replaces a physical object's virtual identity, potentially leading users to harm themselves or others, as when a bystander appears as a game enemy.The authors' selected prior-work examples span puppetry and mismatching attacks; only Optical Marionette mentioned the safety concern of manipulating real-world walking.Source: §5.3.2 · pp. 6–7, §5.3.3 · p. 7

5.4 Characterizing Physical Harm

The authors characterize physical harm by severity, perceived agency, and who causes or receives harm. Their workshop analysis ranges from recoverable discomfort to death and distinguishes harm involving organisms from harm involving non-organisms.

5.4 Characterizing Physical Harm

  • Severity captures how bad physical harm can be and whether the VR user can recover; 16 instances were reported, with falling and punching each appearing four times.The dimension was the most reported in the synthesizing step, appearing in six of 12 dimensions.Source: §5.4 · p. 7
  • Perceived agency measures whether users think they caused the harm; imperceptible VPPM manipulation may make them blame themselves.The dimension ranges from attributing harm to the system to perceiving oneself as responsible.Source: §5.4 · pp. 7–8
  • The harm-origin matrix classifies whether the VR user provokes or receives harm and whether the other party is an organism or non-organism.Some scenarios fit two quadrants; most involve the user receiving harm from a non-organism, while examples also include harm to property, bystanders, pets, and the user from another organism.Source: §5.4 · p. 8

5.5 Observations from the Workshop

Workshop observations connect puppetry attacks with potential ease of deployment, and identify game mechanics and habituation as ways scenarios bring users into harmful actions. A harm matrix organizes who causes or receives harm and the type of other party involved.

5.5 Observations from the Workshop

  • Seven of 19 scenarios used puppetry attacks, sometimes combined with mismatching; the authors suggest these attacks may become early archetypes because they are easier to apply and deploy.Catch a Ride combined false-negative mismatching with walking puppetry.Source: §5.5 · p. 8
  • The harm matrix classifies scenarios by whether the VR user provokes or receives harm and whether the other party is an organism or non-organism.Scenarios with a gray background fit into two quadrants.Source: Figure 5 · p. 8
  • Game narratives and mechanics can direct physical actions, and users may be unaware of what those actions represent in the real world.The authors connect this possibility to the Insult Simulator scenario, in which a user adopts inappropriate postures toward bystanders.Source: §5.5 · p. 8
  • Habituation and trust can be disrupted by adding or removing virtual objects or interrupting an established haptic interaction.In the described scenarios, users become accustomed to an environment or interaction before the manipulation or interruption occurs.Source: §5.5 · p. 8

6 DEMONSTRATING THE POTENTIAL FOR VPPM HARM

The section uses two applications to demonstrate that VPPM techniques can be repurposed to create plausible potential for physical harm. It describes a redirected-walking missing-step scenario and a controller-trajectory scenario, while noting that the applications were not evaluated because of injury risk.

6 DEMONSTRATING THE POTENTIAL FOR VPPM HARM

  • Although workshop-derived attacks may seem impractical, two implementations grounded in prior CHI work demonstrate plausible, pressing potential for harm.The authors note that simple publication-process or platform-level mitigations could counter the applications, but are not currently in place.Source: §6 · pp. 8–9

6.1 SteppingOn: Provoking Missing Steps Using Redirected Walking

  • SteppingOn maps three virtual stairs onto one physical stair, then randomly disables redirection to create a missing step that can trigger a stumble.The application uses imperceptible scene rotation to keep the virtual stairs aligned with the physical stair before turning redirection off.Source: §6.1 · p. 9, Figure 6 · p. 9

6.2 HittingFace: Changing the Trajectory of Controller Movement to Provoke Collision with the HMD

  • HittingFace abruptly offsets the virtual controller as a user moves it toward the HMD, changing its trajectory and provoking a collision.The scenario has the user select baseball caps for their avatar; the offset shifts the virtual controller away from its physical position.Source: §6.2 · p. 9, Figure 7 · p. 9

6.3 Reflection

  • The applications illustrate how existing VPPM concepts can be trivially subverted, but the authors did not evaluate them because of participant-injury risk.The demonstrations are intended to raise awareness and initiate discussion about malicious VPPM use and countermeasures.Source: §6.3 · p. 10

7 MITIGATIONS AND COUNTERMEASURES

The authors propose safeguards spanning research practice, user consent, app platforms, devices, and community guidelines. Their recommendations aim to make VPPM use more transparent and limit unpermitted or abusive implementations.

  • Applications should disclose VPPM use and intent, explain manipulated actions and possible effects, offer user-selected deception levels, and allow users to opt out.The authors recommend informed consent where a VPPM is particularly risky or open to abuse.Source: §7 · p. 10
  • App platforms should verify the type and extent of VPPMs in applications and develop standardized ratings to inform end-users.Source: §7 · p. 10
  • Platform APIs could provide safe, validated VPPMs, while platform-level monitoring could detect movement discrepancies or dangerous virtual-physical overlaps.Source: §7 · p. 10
  • Device-level permission controls and access lists could prevent third-party applications from abusing access to sensory data.Source: §7 · p. 10
  • The authors propose voluntary community guidelines as an immediate route toward regulation, with possible later legal protections for perceptual integrity, agency, and consent.Source: §7 · p. 10
  • Researchers should consider VPPM risks when publishing, favor perceptible manipulations over imperceptible ones, and discuss potential threats when a technique could enable abuse.The paper points to perceptible thresholds as a way to let users know they are being manipulated.Source: §7 · p. 11

8 DISCUSSION

The discussion presents an initial speculative exploration of VPPM risks, identifying potential attacks and dimensions of harm. It also emphasizes that the work maps possibilities rather than establishing how likely attacks or everyday physical harm are.

8 DISCUSSION

  • The workshops identify five potential attacks, a provoke/receive harm matrix, and severity and perceived agency as important dimensions.The attacks are puppetry (walking and arm movement) and mismatching (false-positive, false-negative, and swapping).Source: §8 · p. 11
  • VPPMs may expose users who are disconnected from the real world to motion manipulation, prompting calls for developers to mitigate attacks and prioritize physical safety.The authors note that manipulated physical movements could have effects beyond hitting furniture.Source: §8 · p. 11

8.1 Limitations

  • Because malicious use is difficult to predict before widespread adoption but harder to control once technology is entrenched, the authors use speculative design to critique current practices and consider future implications.Source: §8.1 · p. 11
  • The workshops cannot establish attack likelihood in everyday VR, and their HCI and design participants represent only one perspective rather than safety, security, or technical XR expertise.The authors suggest surveying in-the-wild VR phenomena as one route toward early detection.Source: §8.1 · p. 11

8.2 Future Work

  • The authors describe this as a first exploration and call for further research as XR sensing grows, before real users are harmed.Source: §8.2 · pp. 11–12
  • The paper focuses on physical harm, while workshop participants also raised psychological harms such as phobias and longer-term trauma.Source: §8.2 · p. 12
  • The study examines a narrow range of harms; future work could address multi-user attacks, harm to bystanders, and circumstances that cause harm after a VR session.Workshop scenarios include attacks involving bystanders and pedestrians.Source: §8.2 · p. 12
  • In video see-through headsets, puppetry and mismatching remain possible, while puppetry is more challenging in optical see-through headsets because users can see their physical movements.The authors also identify mismatching as a possible way to provoke falls across see-through headset types.Source: §8.2 · p. 12

9 CONCLUSION

The paper defines VPPMs, uses two speculative design workshops to characterize harmful scenarios and physical harm, and demonstrates misuse through two sample applications. It proposes mitigations and recommendations while warning that published perceptual vulnerabilities may enable future malicious use.

9 CONCLUSION

  • The paper defines VPPMs, identifies puppetry and mismatching attacks, and characterizes physical harm through speculative design workshops.The workshops involved n=8 participants.Source: §9 · p. 12
  • SteppingOn and HittingFace demonstrate how current VPPM research concepts can be trivially subverted for harmful intent.Source: §9 · p. 12
  • The paper proposes platform-level mitigations and preventative recommendations for practitioners and researchers.Source: §9 · p. 12
  • The authors warn that published human-perception thresholds cannot be patched and may provide malicious actors with exploitable vulnerabilities.Source: §9 · p. 12

A DESCRIPTIONS OF SELECTED SCENARIOS

The selected scenarios illustrate how VR applications could redirect movement, alter perceived objects or targets, and shape physical actions in ways that risk harm to users and bystanders.

A DESCRIPTIONS OF SELECTED SCENARIOS

Loading 2202.13200v1…