PaperlayerView exact revision on arXiv

Source-linked AI summary

Extended Reality (XR) Remote Research: a Survey of Drawbacks and Opportunities

Jack Ratcliffe, Francesco Soave, Nick Bryan-Kinns, Laurissa Tokarchuk, Ildar Farkhatdinov

arXiv:2101.08046v1cs.HC

TL;DR

Remote XR research remains less established than in-lab research, despite evidence that remote data collection, recruitment, and unsupervised studies can be viable. This paper surveys XR researchers to identify the approach’s limitations and potential, finding promise alongside unresolved challenges in data collection, system development, recruitment, safety, and hardware.

  • Problem

    XR research is predominantly conducted in laboratories with co-present researchers, while remote XR research has limited literature despite evidence of viability and promise.

  • Method

    The paper analyzes survey results and related literature to examine the boundaries and feasibility issues of remote XR experimentation.

  • Results

    The analysis identifies common remote-research concerns, XR-specific issues involving safety and hardware, and potential benefits including remote data collection and reproducible experimental settings.

  • Takeaways & Limitations

    Remote XR research has potential as a useful approach, but researchers need clearer understanding of its limitations and boundaries before broader deployment.

  • Takeaways & Limitations

    The survey did not include researchers from some sub-communities, including those working with vulnerable populations, so further investigation is needed.

Abstract

from arXiv · show

Extended Reality (XR) technology - such as virtual and augmented reality - is now widely used in Human Computer Interaction (HCI), social science and psychology experimentation. However, these experiments are predominantly deployed in-lab with a co-present researcher. Remote experiments, without co-present researchers, have not flourished, despite the success of remote approaches for non-XR investigations. This paper summarises findings from a 30-item survey of 46 XR researchers to understand perceived limitations and benefits of remote XR experimentation. Our thematic analysis identifies concerns common with non-XR remote research, such as participant recruitment, as well as XR-specific issues, including safety and hardware variability. We identify potential positive affordances of XR technology, including leveraging data collection functionalities builtin to HMDs (e.g. hand, gaze tracking) and the portability and reproducibility of an experimental setting. We suggest that XR technology could be conceptualised as an interactive technology and a capable data-collection device suited for remote experimentation.

1 INTRODUCTION

XR research is increasingly used for creative, social, and psychological experiments, but studies remain predominantly laboratory-based with co-present researchers. Remote XR research is underused despite evidence of feasibility and potential benefits, motivating a survey of researchers’ views.

  • XR includes virtual, augmented, and mixed reality technologies increasingly examined in HCI, social science, and psychology research.
  • Most XR studies take place in laboratories with the researcher and participant co-present.
  • Remote XR research has seen limited adoption despite remote methods’ success in non-XR HCI, social, and psychological research.
  • Existing reports indicate remote XR data collection, recruitment, and unsupervised experimentation are feasible, with some results similar to in-lab findings.
  • The paper surveys 46 XR researchers across 30 questions to characterize remote XR research practice, concerns, and potential benefits.

2 LITERATURE

Prior literature portrays remote XR experimentation as promising but constrained by hardware access, participant recruitment, data quality, and uncertainty about comparability with laboratory studies. Existing evidence includes feasible recruitment and replicated findings, while also documenting important differences across settings and recruitment sources.

  • Benefits and drawbacks of XR experiments: XR experiments offer controlled realism and sensory illusions, while also raising concerns about sickness, cognitive load, novelty effects, embodiment, and ethics.
  • Remote XR experiments: Remote and in-lab VR studies can differ substantially in performance while showing no significant differences between experimental-condition effects.
  • Remote XR experiments: Remote XR recruitment can reach participants at home, but high-end hardware is uncommon and completed data sets may be limited.In one study, 18 of 242 eligible responses had high-end VR systems; another reported that only 15% of participants provided completed data sets.
  • Remote XR experiments: Experiments embedded in existing XR communities can access active users but face programming, external-data, and bespoke-hardware limitations.
  • Supervised vs unsupervised: Unsupervised research generally provides suitable data collection, although supervised studies can produce more effective outcomes in some longitudinal settings.
  • Crowdworkers: Viable?: Crowdworker studies can reproduce historical in-lab results and resemble social-media or campus recruitment, but recruitment sources differ in response rates, attention, quality, honesty, diversity, and effect reproduction.
  • Crowdworkers: Viable?: Crowdworker data quality and non-naivete remain concerns, though attention checks and reputation scores can improve collected data quality.

3 METHODOLOGY

The study combines a survey of XR researchers with inductive thematic analysis. Responses came from 46 researchers across 36 predominantly academic institutions and were independently coded before collaborative theme review.

  • Responses were thematically analysed inductively using Braun and Clarke’s six-phase approach.
  • Two independent researchers conducted coding and theme generation, followed by collaborative review to establish final categories.
  • The survey received 46 responses from researchers at 36 predominantly academic institutions.
  • Respondents were mainly based in Europe and North America and included PhD students, university faculty, academic researchers, students, and corporate researchers.

4 PARTICIPANT XR SETUP RESULTS

Respondents’ XR research was predominantly VR-based and used diverse HMD, controller, sensor, and interaction setups. In-lab experience exceeded remote experience, although planned remote studies were slightly more common than planned lab studies.

  • 37 participants had previously run in-lab studies, compared with 14 who had run remote studies.
  • 24 participants planned remote studies in the next six months, compared with 22 planning lab-based studies.
  • 28 respondents categorised their research as VR-only, while 5 reported AR-only and 10 used both VR and AR.
  • VR research mainly used six-degrees-of-freedom HMD systems, reported by 32 respondents, rather than three-degrees-of-freedom systems or CAVEs.
  • HTC Vive and Oculus devices were the predominant hardware families, used by 25 and 23 respondents respectively, across a wide range of off-the-shelf systems.
  • Embodied interactivity and embodied movement were the most common features, reported by 37 and 35 respondents, while abstract interactivity was reported by 8.

5 THEMATIC ANALYSIS RESULTS

The thematic analysis distinguishes XR studies by whether laboratory settings are vital or merely preferred, and by whether remote participation is supervised. Recruitment is a major remote advantage, but hardware access and participant representativeness constrain that benefit.

  • Study sub-types: In-lab settings are vital when bespoke hardware or unique data collection is required, but preferred when researchers mainly value control and data integrity.
  • Study sub-types: Remote studies are especially suited to natural environments, large participant numbers, or experiments that do not require laboratory benefits.
  • Study sub-types: Remote studies may be encapsulated and unsupervised, with explanations, data collection, and the experiment contained in the software or download process.
  • Recruitment: 27 respondents identified recruitment as a potential remote advantage because it reduces travel and scheduling friction and removes geographic restrictions.
  • Recruitment: Remote recruitment can broaden cross-cultural research without requiring local recruitment networks or laboratory partnerships.
  • Recruitment: XR hardware ownership may limit remote recruitment and reduce representativeness, especially for HMD-based VR and AR studies.
  • Participation: 25 respondents said encapsulated, unsupervised remote studies could reduce scheduling demands and researcher time, though longitudinal studies may experience more dropouts.

5.3 Data Collection

Data collection was the dominant drawback identified for remote XR research. Respondents described challenges involving bespoke hardware, monitoring and sensing, participant burden, data transfer, privacy, and unsupervised behavior.

  • Data collection: The dominant remote XR drawback was data collection, especially bespoke hardware, monitoring and sensing, and data transmission and storage.
  • Bespoke hardware: 13 respondents worried that shipping and configuring bespoke systems would be difficult, especially alongside the demands of PC-based VR and additional recording equipment.
  • Participant burden: Four respondents considered it unreasonable to require remote participants to prepare multiple collection methods, while informal observations such as body language could be lost.
  • Data transmission and storage: Remote studies must encapsulate data capture in the XR experience, while large transfers and sensitive information create reliability and privacy concerns.
  • Monitoring and sensing: Researchers reported that EEG, ECG, eye tracking, hand tracking, galvanic skin response, facial expressions, and body language were often infeasible remotely.
  • Monitoring and sensing: Five researchers explored substitutes such as HMD orientation for eye tracking, microphone-recorded breaths for ECG-based exertion measures, and controller-based hand tracking.
  • Unsupervised participation: Unsupervised remote participation can reduce qualitative feedback and invite response gaming, although it may also reduce bias from researchers’ perceptions of participants.

5.4 Theme: Experiment Processes

Remote XR experiments raise concerns about participant guidance, environments, hardware variability, calibration, and development effort, but respondents also identified benefits including longitudinal engagement, lower costs, and new research questions.

  • Process & Guidance: Twenty-four respondents preferred laboratory guidance because unsupervised participants might misunderstand procedures, assumptions, or target actions.Respondents valued introductions, explanations, ad-hoc guidance, and real-time corrections.
  • Process & Guidance: Remote studies may lose researcher-participant rapport, although one VR project reported deeper ongoing engagement through Discord-based community interaction.The passage also notes that rapport can introduce bias.
  • Environment: Remote environments can introduce distractions, collisions, and unknown variables, although some respondents viewed participants’ homes as less artificial or less interruption-prone than laboratories.Five respondents specifically highlighted the value of deploying XR in participants’ own environments.
  • Hardware and software: Supporting varied remote setups increases labor and development time, while calibration errors may produce faulty experiences or data that are harder to verify and repair.Ten respondents expected more time spent developing remote systems, and respondents described calibration problems as especially labor-intensive to fix.
  • Research questions: Remote XR may enable broader longitudinal engagement, lower experiment costs, and research questions comparing different system setups or hardware.Respondents also suggested that automated processes could improve data quality, though longer development times and equipment logistics remain constraints.
  • Hardware and software: Six respondents reported difficulty reliably measuring physiological information, including hand tracking, despite some consumer hardware supporting hand-tracking capabilities.The passage raises a support and awareness gap around available technologies.

5.5 Theme: Health & Safety

Health and safety were major perceived benefits of remote XR because remote participation avoids shared laboratory hardware, but remote settings introduce risks involving supervision, participant environments, liability, and data governance.

  • Health and Safety: Remote XR can reduce health risks from shared HMDs, controllers, and enclosed laboratory spaces, while also reducing some associated administrative overhead.Six respondents said preparing laboratories and organizing studies could become more time-consuming because of safety and contact-tracing procedures.
  • Health and Safety: Remote participation can be unsafe when studies require rapid movement, distressing materials, or researcher support that is normally available in a controlled laboratory.One AR researcher described an unsupervised study in an inappropriate location as potentially incredibly unsafe.
  • Ethics: Respondents raised ethics concerns about incentivizing risky behavior, responsibility for injuries in participants’ homes, and the embodied risks of XR interventions.The paper specifically discusses liability when participants are injured at home.
  • Ethics: Cross-cultural recruitment may broaden access but raises questions about ethics, data storage, and protection rules across countries with different laws and guidelines.The passage notes that remote VR participation may be concentrated in North America because of the distribution of VR users and sales.

6 COVID-19 IMPLICATIONS

COVID-19 suspended or altered many XR studies because shared hardware and laboratory facilities created infection-control challenges. Respondents anticipated eventual laboratory continuation, but with added sanitation, facility, and participant-selection constraints.

  • COVID-19 Implications: COVID-19 caused many study suspensions, and 30 respondents said it would change their research, including movement toward online surveys.Respondents expected laboratory studies to continue eventually, but with additional sanitizing steps.
  • COVID-19 Implications: Safety concerns extended beyond XR hardware to room airflow and official protocols that could vary by country or institution.These requirements affected the suitability of laboratory facilities for continued studies.
  • COVID-19 Implications: Five respondents worried that participants would avoid laboratory studies because of infection risks and that researchers could be responsible for exposing them.These concerns affected both recruitment recovery and researcher responsibility.
  • COVID-19 Implications: COVID-19-related exclusion could bias participant selection toward people willing to enter small rooms and share equipment.Researchers at high risk, or living with high-risk people, might self-exclude from laboratory studies.
  • COVID-19 Implications: Some laboratories continued operating with face masks, but masks combined with HMDs were cumbersome and not fully protective.The paper notes that laboratories did not all face the same problems and that some continued experiments during this period.

7 DISCUSSION

Remote XR research offers potential benefits but remains constrained by recruitment, technical, data-collection, safety, and experimental-control challenges. Respondents also identified encapsulated, unsupervised experiments as a route toward efficiency, reproducibility, and new hardware-based data collection.

  • Recruitment and participants: Recruiting remote XR participants may increase sample size, diversity, and segmentation, but researchers report difficulty finding participants with suitable high-end HMDs.Proposed responses include dedicated XR crowdworker communities, existing consumer XR communities, and hardware-lending schemes, each with representation, cost, or feasibility concerns.
  • Recruitment and participants: Remote XR applications must accommodate varied devices and computers, while often requiring more setup and less intuitive multi-step procedures than ordinary online studies.These technical and procedural demands are identified as distinct obstacles beyond HMD availability.
  • Data collection: XR hardware can collect body position, head movement, breathing, hand tracking, and HMD-angle data that may support remote study of attention, motivation, engagement, and focus.Respondents also note that remote XR video and qualitative collection can be difficult, while infrastructure for storing large-scale XR data and standardized end-to-end frameworks remains incomplete.
  • Health, safety and Covid-19: Remote XR introduces safety and ethics challenges, including ensuring safe movement environments, avoiding simulator sickness, sanitizing equipment, and maintaining physical distancing.Respondents also raised concerns about masks with HMDs, infection risk, and unclear institutional protocols.
  • Experimental design: Researchers may need to redesign experiments for remote settings when hardware is esoteric, while smartphone-based AR investigations may face fewer concerns.One example replaced an in-lab Hololens study with remote viewing of a prerecorded AR-HMD video.
  • Experimental design: Remote settings add uncontrolled variables such as distractions, participant trust and motivation, and environmental differences, although familiar home settings may reduce some concerns.The cited prior research reports that most remote HMD participants engage at home and predominantly alone.
  • “Encapsulated” experiments: the ideal?: All-in-one experiments that combine the XR experience and data collection could run unsupervised, save time, improve replication and transparency, and support experiment versioning.Shared data-logging techniques and standardized frameworks could further support reusable experimental environments, although encapsulation requires additional development and cannot fully control AR environments.
  • Future research direction: The proposed direction is to understand XR hardware’s data-collection affordances, build supporting frameworks, and design research questions around those affordances inside encapsulated experiences.This would require a shift from predominantly laboratory-oriented research practices.

8 LIMITATIONS

The survey provides broad insight into the XR researcher community, but its respondents do not represent all relevant sub-communities. In particular, vulnerable-population researchers were absent.

  • Scope of the survey: The survey may miss sub-community-specific perspectives because it received no responses from researchers studying vulnerable populations.The authors call for further investigation of these sub-communities.

9 CONCLUSION AND RECOMMENDATIONS

The survey finds that remote XR research could be useful but currently faces limitations in data collection, system development, and participant recruitment. The authors recommend examining participants, result comparability, hardware affordances, and ways to create encapsulated experiments.

  • Conclusion: Remote XR research is viewed as potentially useful, but current limitations concern data collection, system development, and uncertainty around participant recruitment.The authors frame these issues as boundaries that require further investigation.
  • Recommendations: Future research should determine who remote XR participants are, whether they are representative, and how to access a large participant pool.These questions address recruitment scale and participant composition.
  • Recommendations: Researchers should assess how remote XR studies affect results compared with in-lab studies and identify what built-in XR data-collection affordances can support.The recommendation links methodological comparison with understanding the capabilities of XR hardware.
  • Recommendations: The paper recommends reducing barriers to creating encapsulated experiment software so remote XR research can better use reproducible, self-contained experiences.This recommendation follows the paper’s proposed focus on encapsulated experiment software.
  • Reconceptualizing XR research: XR could be reconceptualized as data-collection hardware, with research questions designed around its existing capabilities and suitable investigations conducted in participants’ homes.Because XR can carry the experimental environment with it, the home may serve as a natural research location for suitable applications.
Loading 2101.08046v1…