PaperlayerView exact revision on arXiv

Source-linked AI summary

Security and Privacy Approaches in Mixed Reality: A Literature Survey

Jaybie A. de Guzman, Kanchana Thilakarathna, Aruna Seneviratne

arXiv:1802.05797v3cs.CRcs.CYcs.HC

TL;DR

MR research has prioritized enabling the technology, while security and privacy implications remain comparatively underinvestigated. This survey synthesizes protection approaches across MR and related technologies, finding uneven coverage of privacy properties and a need for further integrated mechanisms.

  • Problem

    Only a small share of MR-related research discusses security and privacy impacts or presents protection approaches, despite growing device availability and sensing capabilities.

  • Method

    The survey derives thirteen security and privacy properties from three established models, categorizes approaches across five protection aspects, and reviews related AR, VR, HCI, mobile, wearable, and IoT work.

  • Results

    The reviewed approaches primarily target confidentiality, authorization, and undetectability, while input-protection methods can provide strong confidentiality only within specific target scopes.

  • Takeaways & Limitations

    Security and privacy mechanisms should be designed, investigated, and implemented for integration with existing and upcoming MR systems before their use becomes widespread.

  • Takeaways & Limitations

    Output protection remains constrained by the difficulty of specifying and effectively enforcing reference policies, including deciding who specifies them.

Abstract

from arXiv · show

Mixed reality (MR) technology development is now gaining momentum due to advances in computer vision, sensor fusion, and realistic display technologies. With most of the research and development focused on delivering the promise of MR, there is only barely a few working on the privacy and security implications of this technology. This survey paper aims to put in to light these risks, and to look into the latest security and privacy work on MR. Specifically, we list and review the different protection approaches that have been proposed to ensure user and data security and privacy in MR. We extend the scope to include work on related technologies such as augmented reality (AR), virtual reality (VR), and human-computer interaction (HCI) as crucial components, if not the origins, of MR, as well as numerous related work from the larger area of mobile devices, wearables, and Internet-of-Things (IoT). We highlight the lack of investigation, implementation, and evaluation of data protection approaches in MR. Further challenges and directions on MR security and privacy are also discussed.

1 INTRODUCTION

MR combines sensing, interaction, and realistic displays across the reality–virtuality continuum, but security and privacy research has lagged behind its technological development. This survey situates MR and related AR/VR work while showing that only a small fraction of literature addresses substantive security and privacy concerns.

  • MR Technology and Scope: MR systems combine advances in computer vision, sensor fusion, artificial intelligence, 3D rendering, optics, and display technologies to support immersive interaction with physical and virtual objects.These capabilities enable interactions such as gestures in the air and outputs mixed with the real-world environment.
  • MR Technology and Scope: AR, VR, and augmented virtuality occupy different positions on the reality–virtuality continuum, from augmenting the physical world to integrating real objects into virtual environments.MR is used broadly for devices and displays spanning this continuum, while its current definition also combines aspects of AR and VR.
  • Security and Privacy Motivation: MR devices raise privacy and security risks because wearable systems commonly use cameras to capture environments that may reveal sensitive information about users and bystanders.The paper connects these risks to demonstrated re-identification of people from facial images and publicly available social-network photos.
  • Security and Privacy Motivation: Only 1.42% of Scopus-indexed papers in 2016 substantively discussed security and privacy impacts or approaches for AR, VR, or MR, despite keyword-related papers increasing from 0.7% in 1997 to 5.8% in 2016.The authors supplemented Scopus with searches of Google Scholar, IEEE Xplore, ACM Digital Library, and specialized venues.
  • Prior Work and Survey Scope: Earlier surveys emphasized MR taxonomies and enabling technologies, while related work also addressed ethical issues, value-sensitive design, and protection of input, data access, and output.This survey expands coverage across MR platforms and setups, including non-wearable systems.

Contributions

This survey organizes mixed-reality security and privacy research through a data-centric framework, combining thirteen properties and five protection aspects. It reviews related work, maps approaches to protection properties, and excludes network security while identifying overlapping categories and open directions.

  • Contributions: The survey presents a data-centric categorization of mixed-reality security and privacy work organized into five major aspects and subcategories.It also uses generic system block diagrams to represent different protection mechanisms.
  • Contributions: It broadens the review beyond MR to related security and privacy research from AR, VR, HCI, mobile devices, wearables, and IoT.The survey includes work not necessarily directed at MR when it is related to MR security and privacy considerations.
  • Contributions: The survey identifies target security and privacy properties and summarizes how protection strategies are distributed across them.These properties are derived from three models and combined into an overarching set of thirteen properties, including security and privacy requirements that may coexist or conflict in priority.
  • Contributions: The survey does not focus on network security and instead relies on existing protections for communication networks and data transmission.This defines a scope boundary for the reviewed security and privacy approaches.
  • Contributions: The five protection aspects follow data flow through MR systems, covering sensitive inputs, processing, outputs, interactions, and devices.The categorization is not exclusive: many approaches can belong to multiple categories or subcategories.

3 SECURITY AND PRIVACY APPROACHES

The survey organizes MR security and privacy approaches around intermediary protection layers, input threats, and sanitization strategies. It also covers physical and policy-based controls for limiting capture and adapting protection to sensitive objects and real-time video.

  • Intermediary protection layers control data flows between device resources and applications, especially untrusted third-party services.These layers can enforce limited or less-privileged access to inputs and other resources.
  • Input Protection: Unauthorized or unintended disclosure, detectability, and user content unawareness are central threats because MR sensors capture both necessary and sensitive information.
  • Input Protection: Input sanitization removes latent sensitive information through an intermediary layer that also acts as input access control.Policies may be intrinsic, imposed by the user, device, or system, or extrinsic, addressing sensitive external objects.
  • Input Protection: Protection strategies reduce information, block it completely, or replace raw captures with less revealing representations such as skeletal data.Figure 7 illustrates facial-outline reduction, complete blocking, and skeletal hand information instead of raw video.
  • Input Protection: User-defined and context-based systems vary sanitization detail, blur sensitive features, delete sensitive locations, or sanitize live video feeds.Darkly applies multi-level feature sanitation, while context-based approaches identify faces, license plates, or sensitive locations automatically.
  • Input Protection: Physical and communication-based controls can deter unauthorized capture or enforce fine-grained permissions for continuous sensing.Examples include projector-based blinding of unauthorized devices and policies that blur or deny access to captured objects.
  • Input Protection: Least-privilege input protection remains limited by the need to recognize application requirements and future sensitive objects dynamically.Prepose addresses future gestures but not future objects, while extrinsic markers such as MarkIt address object visibility.

Protected Data Collection and Aggregation.

The survey reviews privacy-preserving collection, aggregation, and sensor-access controls for data gathered by MR-related systems. These approaches address disclosure risks by limiting application access or protecting aggregated information.

  • Privacy-preserving collection and aggregation include randomized response, differential privacy, and aggregation methods for distributed sensor systems.The surveyed examples target crowd-sourced information, wireless sensor networks, and body area networks.
  • SemaDroid extends Android sensor management with fine-grained user control over application access to sensors and sensor data.It operates as an intermediary protection layer providing application authorization for sensor resources.
  • During processing, third-party services may directly access sensitive data, exposing it to disclosure, linkability, detectability, and identifiability threats.
  • Secure and privacy-preserving processing algorithms can provide properties including integrity, confidentiality, unlinkability, and plausible deniability.

Protection Approaches.

The survey describes encryption, secret sharing, secure multi-party computation, and sanitized reconstruction as ways to process or share MR-related data without exposing complete sensitive inputs. These methods trade protection, flexibility, computation, or reconstruction latency against application utility.

  • Homomorphic encryption enables third-party computation over encrypted data without decryption, supporting remote processing when data processors are untrusted.The survey distinguishes partial schemes such as Paillier encryption from fully homomorphic encryption.
  • Visual encryption methods protect image features before extraction, matching, or context inference, while later variants reduce computation time or broaden supported tasks.He-Sift protects dominant features, whereas SecSIFT supports color-histogram and layout-descriptor processing.
  • Secure multi-party computation and secret sharing combine data from multiple sources without requiring parties to reveal their actual data shares.Figure 8 depicts encrypted shares exchanged between parties to derive combined knowledge.
  • Encryption-based protection can be constrained by format-specific assumptions, including an approach described as JPEG-format specific.
  • Privacy-preserving services can split sensitive information between devices and servers, as in secret-shared anthropometric data for virtual cloth try-on.
  • Sanitized 3D reconstructions can replace complete RGB-depth scenes with generalized surfaces while balancing privacy protection against reconstruction latency.The survey connects this approach with differential privacy, k-anonymity, sanitization, abstraction, and possible honeypot defenses.

Data Storage Solutions to Protection.

Protected storage approaches give users more control over data access through personal data stores, sandboxing, trusted applets, and monitored data flows. The survey notes that most reviewed techniques were not implemented specifically for MR and may require substantial resources or application changes.

  • Data Storage Solutions to Protection: Personal data stores vary in flexibility: SafeAnswers uses pre-approved queries, while Databox uses trusted containerized applications in a sandbox.These requirements may hinder adaptation, although Databox also frames data exchange as a privacy ecosystem.
  • Data Storage Solutions to Protection: Most reviewed data-protection techniques target generic systems rather than MR and may require application changes, memory, and computation resources.The survey identifies combining protected sensor management, sanitization, and authorized storage access as promising.
  • Output Protection: MR protection must also cover rendered outputs because tampering, spoofing, denial of service, and consent non-compliance can compromise reliability and user safety.Output protection includes external-display protection, output control, and protected rendering.
  • Output Protection: Output access-control policies can govern rendering priority through properties such as transparency, arrangement, and occlusion.

Safe and Reliable Outputs.

Safe MR outputs limit applications to the display information they need while mediating rendering through output policies. This protects sensitive content in physical or detected display media and includes Arya as a prototype approach.

  • Safe and Reliable Outputs.: Arya implements application output control from specified output policies and is evaluated in simulated scenarios.The survey identifies Arya as the only AR or MR output access-control approach in the literature at that point.
  • Safe and Reliable Outputs.: MR rendering can expose latent or sensitive information in physical surfaces while detecting suitable display media.A wall used as a display surface may reveal objects or other information embedded in the wall during detection.
  • Safe and Reliable Outputs.: Protected output rendering reveals only surface structure, location, and orientation instead of the sensitive contents of detected display media.The approach applies least privilege through an intermediary layer between applications and rendering resources.

Privacy-preserving Rendering.

MR can protect confidential outputs by using dedicated output-defense strategies, but output displays remain vulnerable to physical inference and visual-channel attacks.

  • Privacy-preserving Rendering.: Output displays remain vulnerable to physical inference and visual-channel attacks such as shoulder surfing.These threats also arise for user inputs, especially when input and output interfaces share or integrate the same medium.
  • Privacy-preserving Rendering.: MR can provide output confidentiality for sensitive contexts such as ATM transactions by leveraging its capabilities for output defense.

Protecting outputs from external inference.

MR output protection addresses observation and capture by hiding content, sharing visual secrets, and mediating access to rendering resources. These approaches protect against human viewing in some settings but remain subject to machine-aided capture and policy-enforcement challenges.

  • Protecting outputs from external inference.: EyeGuide uses a near-eye display to deliver secret navigation information over a public map, preventing shoulder surfing because the display is practically secret.
  • Protecting outputs from external inference.: VRCodes hide codes from human eyes through rolling-shutter timing, but cameras operating at a specific frame rate can still detect them.Related techniques similarly hide AR tags in video while remaining vulnerable to machine-aided inference or capture.
  • Protecting outputs from external inference.: Visual secret sharing decrypts content by overlaying a visual cipher with a visual key, while code-based schemes relax the strict alignment required by classical VSS.The cited code-based forms include barcodes, QR codes, and 2D barcodes.
  • Protecting outputs from external inference.: Visual cryptography and content-hiding methods provide visual access control and protect information on shared displays and during input sensing.Captured ciphers remain secure when the corresponding secret shares or keys are protected.
  • Protecting outputs from external inference.: Output abstraction can mediate access between applications and rendering resources, but specifying and enforcing the reference policies remains a major challenge.The survey emphasizes that output-side risks are especially immediate because malicious responses may already be actuated.

Protecting Collaborative Interactions.

Collaborative MR protects interactions through user-specified policies, privacy-aware shared and private spaces, feed-through signals, and secure channel initialization. These mechanisms address information sharing, privacy negotiation, and unauthorized access across collaborative environments.

  • Protecting Collaborative Interactions.: User-originated policies and enforcement are primary strategies for protecting collaborative interactions.The interaction system must support both policy specification and policy enforcement.
  • Protecting Collaborative Interactions.: Feed-through signalling helps users negotiate privacy preferences by providing information about shared-space entry and other participants' activity.Examples include alarms for users entering a shared space, gesture-based private sessions, and visual indicators of activity.
  • Protecting Collaborative Interactions.: Feed-through mechanisms can still pose security and privacy risks because their pervasive signals may expose information in shared environments.
  • Protecting Collaborative Interactions.: Collaborative MR uses shared spaces for public objects and private spaces for user-sensitive tasks such as strategy formation.Private Interaction Panels provide a private region while users continue viewing public content through their head-mounted displays.
  • Protecting Collaborative Interactions.: Sharing initialization must address spoofing and unauthorized access when no pre-existing secure channel or supporting architecture is available.Out-of-band approaches such as LGTM combine wireless distance information and facial recognition to cross-authenticate users.

Securing Sharing Channels.

MR sharing protections must be chosen according to the use case because privacy concerns and acceptable disclosure vary across contexts. Device protection therefore addresses both access control and display leakage.

  • Securing Sharing Channels.: Protection choices are context-dependent: feed-through signalling may help in classrooms but feel invasive during military negotiations.The paper frames protection as a subjective decision about who or what is being protected.
  • Securing Sharing Channels.: Device-level protection covers two aspects: device access and display protection.The survey treats device protection as complementary to broader data-protection approaches.
  • Securing Sharing Channels.: Device access control aims to admit authorized users and bar unauthorized ones, addressing identity spoofing and unauthorized access.The paper identifies identity spoofing and unauthorized access as primary device-access threats.

Novel Authentication Strategies.

The survey reviews gesture, physiological, multimodal, optical, and visual-cryptographic strategies for authenticating users and protecting MR interfaces. These approaches improve confidentiality but remain constrained by alignment requirements and narrow sensitive-use cases.

  • Novel Authentication Strategies.: Mid-air finger and hand gestures achieved 86-91% accuracy across 200 users using a 3D camera-based motion controller.The reported accuracy is based on corresponding equal error rate measurements.
  • Novel Authentication Strategies.: Multimodal authentication combining facial, iris, and periocular information achieved a 0.68% EER, while gaze-and-touch passkeys target shoulder surfing.These methods combine multiple biometric or interaction signals for authentication.
  • Novel Authentication Strategies.: Optical protection uses polarization, narrowband illumination, or both to preserve display transmission while reducing leakage.The survey identifies this as the only reported MR display-leakage protection work using optical strategies at that point.
  • Novel Authentication Strategies.: Visual cryptography lets an AR device decrypt an encrypted public interface so only the user sees the actual input layout.EyeDecrypt keeps the public interface encrypted, while a user-held secret key enables visual decryption through the AR display.
  • Novel Authentication Strategies.: Visual-cryptography approaches remain limited by physical-display alignment and are mainly applicable to predetermined secrets such as passwords or ATM PINs.The paper notes that alignment requirements confine usability to specific sensitive activities.
  • Novel Authentication Strategies.: The surveyed approaches target confidentiality, but protection may be significant only within a specific target space or entity.PrePose hides raw input while exposing gesture events, and Recognizers exposes recognizer data while hiding raw RGB and depth information.

Generalizations and gaps.

Across the surveyed approaches, confidentiality, authorization, and undetectability are common targets, while unlinkability and plausible deniability remain under-addressed. The distribution of protections varies by category, with input approaches leaning more toward privacy and device approaches toward security.

  • Generalizations and gaps.: Input approaches commonly provide authorization, undetectability, policy compliance, anonymity, and content awareness, but none target unlinkability.Access control and abstraction can cease to preserve anonymity and undetectability once applications receive the underlying media or abstractions.
  • Generalizations and gaps.: Only Rappor and privacy-preserving virtual cloth try-on combine anonymity, undetectability, and unlinkability through statistical privacy techniques.Rappor uses randomized response for differential privacy during data collection.
  • Generalizations and gaps.: Output protection primarily targets undetectability through visual hiding or cryptography, while output policies support availability, reliability, and user safety.Abstraction strategies also provide policy-compliant, authorized access to output and rendering resources.
  • Generalizations and gaps.: Interaction protections emphasize awareness and policy or consent compliance, while sharing initialization addresses non-repudiation, authentication, identification, and integrity.The survey distinguishes user-mediated information release and feed-forwarding from sharing-initialization strategies.
  • Generalizations and gaps.: Device protections primarily target authorization, identification, and confidentiality, while visual protections prevent unauthorized inference from displays and devices.The category-level comparison indicates that the first three categories lean toward privacy and device protection leans toward security.
  • Generalizations and gaps.: Unlinkability and plausible deniability remain the clearest gaps, while authorization and undetectability are among the most targeted properties.Figure 11 shows that data-protection approaches, mostly generic rather than MR-specific, primarily target the two under-addressed properties.

4 OPEN CHALLENGES

The survey identifies open MR challenges spanning security and privacy evaluation, platform support, targeted data protection, regulation, networking, interoperability, and user welfare.

  • Security and Privacy of Existing Devices and Platforms: MR platforms and devices still lack systematic security and privacy evaluation, leaving latent risks insufficiently identified and protections untested.The survey specifically calls for analysis of applications, devices, and platforms, noting that prior work on Wikitude and Layar lacked systematic evaluation.
  • Regulation and User Welfare: MR applications raise regulation and welfare concerns involving vulnerabilities, bystander privacy, information correctness, safety, legality, and physical or cognitive burden.The survey links sensing and rendering capabilities to regulatory concerns and identifies discomfort, occlusion, focus, jitter, and reduced tactile feedback as user-welfare issues.
  • Native Support: Monolithic MR applications retain broad access to sensing and rendering resources, motivating operating-system support with finer-grained access control.Existing native support, including ARKit, is described as coarse-grained and still exposed to similar risks.
  • Targeted Data Protection: MR needs targeted data protection for collection, processing, and storage because its rich, dynamic data may not map readily onto generic protections.Suggested approaches include encryption, privacy-preserving algorithms, and personal data stores designed for MR data.
  • Interoperability and Networking: Widespread MR adoption also depends on interoperability and networks capable of transporting, processing, and rendering data at acceptable latency.The survey notes that current communication data rates may not meet the expected demand and that network infrastructure may need redesign.
  • Smart Future: A future smart MR ecosystem remains constrained by processing, storage, communication, energy-management, and battery challenges.

5 CONCLUSION

The survey consolidates MR security and privacy risks and approaches into five protection aspects, then compares them across thirteen properties. It finds that confidentiality, authorization, and undetectability receive the most attention, while several privacy properties remain underprovided.

  • Survey Scope and Organization: The survey collects, categorizes, and reviews MR security and privacy approaches across input, data access, output, interactivity, and device integrity.It extends coverage to related HMDs, software platforms, toolkits, and APIs.
  • Comparative Framework: The comparison uses thirteen security and privacy properties derived from three established models to characterize the reviewed approaches.
  • Findings: Confidentiality, authorization, and undetectability are the most targeted properties, whereas unlinkability and plausible deniability receive comparatively little provision.
  • Future Direction: The survey concludes that security and privacy mechanisms should be designed, investigated, and implemented for integration with existing and upcoming MR systems.The recommendation is framed while MR utilization and adoption remain not widespread.

C SUMMARIZING SECURITY AND PRIVACY STRATEGIES

The survey compares security and privacy strategies using generic control families and finds uneven coverage across input, data, output, interaction, and device protection.

  • Comparison Framework: The comparison organizes reviewed approaches by security and privacy controls, using eleven generic control families based on NIST controls.
  • Input Protection: Input protection commonly uses media sanitization, media protection, and access policies, while only two approaches apply resource access policies and least privilege.
  • Data Protection: Only one reviewed approach was applied in an MR context for data protection, leaving collection, processing, and storage insufficiently addressed.The survey identifies probabilistic controls as another underexplored possibility.
  • Output Protection: Output strategies variously protect shared resources, physical interfaces, transmission, cryptographic content, and application access to displays.
  • Interaction Protection: Interaction protections address information sharing in shared spaces and commonly use distributed processing controls.
  • Device Protection: Device protection is the only category using device authentication controls, with the last five approaches emphasizing physical device access.
Loading 1802.05797v3…