PaperlayerView exact revision on arXiv

Source-linked AI summary

GenXR-Bench: Benchmarking Coding Agents for Interactive XR Development

William Huang, David Li, Xiang 'Anthony' Chen, Xun Qian, Nels Numan, Zhongyi Zhou, Hyunsung Cho, Salman Muin Kayser Chishti, Ding Xia, Xincheng Huang, Jiahao Ren, Faraz Faruqi, Tianyu Xu, Yang Zhang, David Kim, Ruofei Du

arXiv:2610.29014v1cs.AIcs.GRcs.HC

TL;DR

XR coding agents are difficult to evaluate because runtime correctness depends on continuous user motion, spatial input, and environmental sensing. GenXR-Bench pairs expert-authored XR tasks with interactive testing in simulated environments. The strongest configuration achieves 50.9% overall Pass@1, while the authors identify simulator limitations in capturing physiological and accessibility constraints and noisy sensor input.

  • Problem

    XR software evaluation must account for continuous user motion, spatial input, and environmental sensing, which conventional software evaluation does not capture.

    Source: §1 · pp. 1–2

  • Method

    GenXR-Bench provides 112 expert-curated XR tasks and held-out tests combining scripted and adaptive actors with runtime assertions and visual checks.

    Source: §1 · p. 2, §3.4 · pp. 6–7

  • Results

    The strongest configuration achieves 50.9% overall Pass@1, while frontier agents continue to struggle with XR development tasks.

    Source: §1 · p. 2, §4.2 · p. 8

  • Takeaways & Limitations

    GenXR-Bench provides a reproducible testbed for evaluating coding agents in physically situated spatial-computing conditions.

    Source: §5 · p. 10

    The simulator does not capture physiological and accessibility constraints, and its perfect sensor input represents a best-case evaluation setting.

    Source: I LIMITATIONS · p. 32

Abstract

from arXiv · show

Coding agents are rapidly advancing from single-file generation to long-horizon repository-scale engineering, yet the evaluation of their capabilities to author interactive extended reality (XR) applications remains an open challenge. Unlike traditional software, XR applications operate within dynamic 3D physical environments where runtime robustness depends jointly on continuous user motion, spatial input, and environmental sensing. We introduce GenXR-Bench, the first benchmark for the end-to-end evaluation of coding agents in XR software development. GenXR-Bench contains 112 expert-authored tasks spanning 24 feature implementations, 21 bug repairs, and 67 application generation tasks. Held-out tests combine scripted and adaptive interactions with programmatic and visual assertions to evaluate core functionality and runtime behavior. GenXR-Bench's testing suite achieves 96.5% accuracy against blinded expert judgments of 124 candidate applications executed on consumer XR hardware. Evaluating 10 models on GenXR-Bench, we find that the strongest configuration achieves only 50.9% overall Pass@1. Failure analysis identifies recurring misunderstandings in asynchronous sensor lifecycles, physical grounding, and embodied interaction semantics. GenXR-Bench establishes a reproducible testbed for evaluating coding agents in the interactive and physically situated conditions of spatial computing.

1 INTRODUCTION

XR development requires coding agents to reason over continuous physical motion, spatial input, and environmental sensing, creating evaluation challenges that existing software benchmarks do not address. GenXR-Bench and GenXR-Test provide a task benchmark and runtime evaluation suite; validation against device-based expert judgments and model evaluations show both reliable testing and persistent agent difficulties.

1 INTRODUCTION

  • XR evaluation must account for continuous 3D motion, spatial transformations, sensing, rendering, and physical ergonomics, unlike discrete 2D event-loop software.Existing unit tests, browser runners, and static 3D generation benchmarks do not verify the full combination of spatial scenes and continuous tracking inputs.Source: §1 · pp. 1–2
  • GenXR-Bench covers 112 tasks: 24 SDK feature implementations, 21 bug repairs, and 67 end-to-end application-generation tasks.GenXR-Test combines deterministic kinematic actors and adaptive VLM-controlled agents with scene-graph assertions and scoped visual checks.Source: §1 · p. 2
  • 50.9% overall Pass@1 is the strongest configuration’s result; 96.5% mean accuracy against expert judgments and 4.0% flakiness validate the automated suite.The validation used 124 applications across 372 evaluation runs on consumer XR hardware.Source: §1 · p. 2
  • Failure analysis identifies coordinate-transformation errors, asynchronous sensor initialization races, and confusion about bimanual interaction semantics as recurring bottlenecks.Across failed runs, 64.4% exhibited coordinate-transformation errors, 44.9% encountered sensor initialization races, and 34.5% failed to distinguish bimanual semantics.Source: §1 · p. 2

2 RELATED WORK

Prior work evaluates interactive software, language-driven 3D and XR authoring, and automated XR testing, but GenXR-Bench focuses on verifying task-specific functional and interactive requirements in XR applications.

2.1 INTERACTIVE SOFTWARE ENGINEERING BENCHMARKS

  • Interactive software benchmarks use browser actions, multimodal rendering judgments, or game interaction tests; GenXR-Bench extends runtime evaluation to embodied interaction and environmental sensing.It formalizes evaluation of physically grounded interactive applications beyond reliance purely on LLM judges.Source: §2.1 · p. 3

2.2 GENERATIVE 3D AND XR AUTHORING

  • Language-driven 3D and XR systems generate scenes, layouts, features, behaviors, or structured specifications, while GenXR-Bench checks task-specific functional and interactive requirements.It builds on XR Blocks, which prior work used to author XR applications that initialize without errors (Du et al., 2026).Source: §2.2 · pp. 3–4

2.3 AUTOMATED XR TESTING

  • Prior XR testing includes replayed inputs, interface heuristics, scene-graph tests, simulated actions, and interaction-flow graphs; GenXR-Test enables deterministic, implementation-agnostic functional tests in a reproducible simulator.The cited work also includes runtime conformance testing and commercial LLM-driven XR testing tools.Source: §2.3 · p. 4

3 GENXR-BENCH

GenXR-Bench packages XR engineering and application-generation tasks with held-out tests in a shared XR Blocks scaffold. Its task set spans three development workflows, and GenXR-Test combines simulated interaction with runtime checks and a separate application-design score.

3 GENXR-BENCH

  • The 112 tasks each pair a natural-language specification and editable project scaffold with a held-out, human-authored evaluation suite.All tasks use XR Blocks, providing a common framework and reducing variation from engine-specific editors and deployment workflows.Source: §3 · pp. 4–5

3.1 BENCHMARK COMPOSITION

  • Feature tasks test implementation of missing XR Blocks capabilities, while repair tasks require fixing a reproduced behavioral defect without breaking related functionality.Feature tasks retain the surrounding SDK, interfaces, documentation, and examples; repair specifications describe observable behavior without naming the faulty implementation.Source: §3.1 · p. 5

3.2 BENCHMARK CURATION AND CONSTRUCTION

  • SDK tasks derive from maintainer reports, pull requests, and issues, while application tasks were selected from a taxonomy of 436 XR systems papers and expanded into full specifications.Application specifications and tests were refined using expert-reviewed valid and adversarial candidates, accommodating multiple valid implementations.Source: §3.2 · pp. 5–6

3.3 BENCHMARK STATISTICS

  • GenXR-Bench contains 112 tasks: 24 SDK feature implementations, 21 bug repairs, and 67 application-generation tasks.Experts rated 28 tasks easy, 38 medium, and 46 hard; tasks can carry multiple XR capability labels.Source: §3.3 · p. 6

3.4 GENXR-TEST EVALUATION FRAMEWORK

  • GenXR-Test combines actors that simulate user and environmental inputs with assertions that check application behavior in a custom simulator with articulated hands and physical scenes.Scripted actors support reproducible predefined actions; adaptive VLM-powered actors handle actions that depend on the candidate implementation. Programmatic assertions inspect runtime properties, while scoped VLM assertions check visual outcomes.Source: §3.4 · pp. 6–7

3.5 EVALUATING APPLICATION DESIGN

  • Application-generation tasks receive a separate design score: a VLM rates recorded application videos against ten XR design heuristics on a 1–5 Likert scale.The score supplements functional evaluation rather than replacing it.Source: §3.5 · p. 7

3.6 ANALYSIS OF APP VERIFIER RELIABILITY

  • Across 372 evaluation runs, the verifier aligned strongly with human evaluations (96.5% accuracy) and was reproducible (4.0% flakiness); disagreements primarily involved interaction behaviors the automated tests did not accommodate.Source: §3.6 · p. 8

4 EVALUATIONS

Across the evaluated configurations, frontier models still struggled with XR development, and success varied across task categories and design quality. Failures commonly involved asynchronous runtime state, spatial coordinate frames, and embodied interaction.

4.1 EVALUATION SETUP

  • The evaluation used ten model configurations, a mini-swe-agent harness within Harbor, and Pass@1 for functionality; application-generation tasks also received a mean Design Score.Agents had access to XR Blocks source, documentation, examples, and relevant dependency documentation, with other network access restricted.Source: §4.1 · p. 8

4.2 DISCUSSION OF RESULTS

  • 50.9 Pass@1 was the best overall result, while claude-opus-5 achieved 47.8% application-generation Pass@1 and gpt-5.6-sol achieved a 4.03 mean Design Score.SDK repair scores were high, but SDK implementation and application-generation scores remained low; models more willing to evaluate code through runtime tests were also more successful.Source: §4.2 · pp. 8–9

4.3 FAILURE ANALYSIS AND DIRECTIONS FOR IMPROVEMENT

  • Startup race conditions occurred in 44.9% of failed runs, as implementations used assets or tracking state before availability or failed to respond to later surface detection.Some applications that passed tests still had limited functionality because they checked for detected surfaces only at startup.Source: §4.3 · p. 9
  • Coordinate-frame errors occurred in 64.4% of failed runs, often when implementations compared positions relative to different parent objects without transforming them to a common frame.A recurring example was placing buttons on a hand’s root joint rather than on the palm surface.Source: §4.3 · p. 9
  • Hand interactions and asymmetric gestures accounted for 34.5% of failures; models also mishandled gesture recognition, continuous physical contact, and ergonomic interface placement.Some implementations treated touch as a discrete click using Boolean bounding-box intersections, producing unstable contact detection.Source: §4.3 · pp. 9–10

5 CONCLUSION

GenXR-Bench evaluates functional and interactive correctness in agent-generated XR software through 112 expert-curated tasks and GenXR-Test. The authors find that frontier agents still struggle with embodied interactions and asynchronous runtime state.

5 CONCLUSION

  • GenXR-Bench is a benchmark of 112 expert-curated XR software engineering tasks, built with the GenXR-Test automated XR testing framework.Five XR experts curated tasks spanning core XR engineering workflows within XR Blocks.Source: §5 · p. 10
  • Frontier coding agents still struggle with XR tasks that require continuous reasoning over embodied interactions and asynchronous XR runtime state.The authors identify improving agents’ embodied understanding as a need.Source: §5 · p. 10
  • The authors envision GenXR-Bench as a testbed for advancing spatially situated software engineering across spatial AI, perception, and embodied interactions.Source: §5 · p. 10

6 AI USE DISCLOSURE

The authors used generative AI tools across several research and writing activities, while domain experts guided and reviewed AI-assisted work.

6 AI USE DISCLOSURE

  • Generative AI tools supported dataset generation, benchmark authoring, data analysis, and paper writing and editing, but not conceptual-framework development.The authors state that domain experts guided the work and reviewed AI-assisted work, and take responsibility for the final content and artifacts.Source: §6 · p. 10

7 ETHICS STATEMENT

The work studies LLM-based code generation for XR applications under a deliberately limited, safety-conscious scope.

7 ETHICS STATEMENT

  • The study focuses on how models reason about spatial computing and implement it in XR applications.Its data collection was limited to real-world XR frameworks and guided by contributors to keep data controlled and accessible.Source: §7 · p. 10
  • The authors limit the work to safe, non-sensitive settings because XR applications can expose users to physical harm by obscuring real-world hazards.Source: §7 · p. 10
  • The authors report anonymizing potentially personally identifiable information and using publicly available pretrained models in accordance with their licenses.Source: §7 · p. 10

8 REPRODUCIBILITY STATEMENT

The authors describe steps intended to make the benchmark and its results reproducible, including releasing prompts and documenting construction and evaluation.

8 REPRODUCIBILITY STATEMENT

  • All prompts used for data generation and evaluation are published in the appendix, alongside details of dataset construction, preprocessing, and evaluation metrics.Source: §8 · p. 10
  • The benchmark is slated for open-source release upon publication.Source: §8 · p. 10
  • The benchmark is implemented using the Harbor Framework to support reproduction and access.Source: §8 · p. 10

A GENXR-BENCH IMPLEMENTATION

The implementation combines runtime interaction testing with a curated process for generating application prompts and XR-specific usability heuristics. The appendices also show examples of feature, repair, and application-generation tasks.

A.1 RUNTIME VERIFIER IMPLEMENTATION

  • Each test session is deterministic and replayable across dominant hands and physical scenes, records video, and uses semantic targets to improve reproducibility and reduce adaptive-actor steps.The verifier runs applications through Playwright with Vitest or a custom Vitest extension; its session injects runtime readers and action controllers for state assertions and embodied actions.Source: §A.1 · pp. 18–19

B XR APP PROMPT CREATION

  • The 67 application ideas were drawn from a taxonomy built by two XR experts from 436 peer-reviewed XR systems papers, then generated, screened, expanded, and manually edited.Evaluators generated 1,000 candidate ideas; an XR expert screened them for XR relevance and feasibility before selecting the final ideas.Source: B XR APP PROMPT CREATION · pp. 20–21
  • Application specifications cover capabilities, required physical objects and settings, step-by-step user experience, evaluation tasks, and implementation difficulty.Evaluation criteria include positive interactions and cases such as missing objects, lost tracking, invalid gestures, and unexpected inputs.Source: B XR APP PROMPT CREATION · pp. 21–22

C XR HEURISTICS

  • XR usability heuristics were derived from formal XR application evaluations, adapted from Nielsen’s 10 heuristics, and filtered to exclude hardware-specific or human-cognitive-testing measures.The authors analyzed 474 papers, including 38 additional papers on formal XR evaluations, and phrased the resulting heuristics for 1–5 Likert scoring.Source: C XR HEURISTICS · p. 22

D EXAMPLES OF GENXR-BENCH TASKS

E ANALYSIS OF XR APP JUDGE ALIGNMENT

The VLM judge’s ratings align with three-rater expert consensus on XR design and usability heuristics, reaching agreement levels that match or exceed individual human raters. The evaluation uses 124 applications rated by XR experts.

E ANALYSIS OF XR APP JUDGE ALIGNMENT

  • Eight XR experts rated the same 124 applications generated by two Gemini models across 62 app tasks using the paper’s XR heuristics.Because the evaluation was large, the same three evaluators did not rate every candidate, making traditional inter-rater agreement statistics inapplicable.Source: E ANALYSIS OF XR APP JUDGE ALIGNMENT · p. 24
  • Individual-rater agreement was low, which the authors attribute to XR applications being complex and time-consuming to evaluate.The three-rater consensus was nevertheless reported as reliable.Source: E ANALYSIS OF XR APP JUDGE ALIGNMENT · p. 25
  • 35.30% exact-score agreement and 71.10% agreement within one point for the VLM judge, versus 29.80% and 65.60% for human-to-human agreement, respectively.Three-rater consensus reliability was IRR3 = 0.61; the table reports agreement across design heuristics for 124 applications.Source: Table 7 · p. 25, E ANALYSIS OF XR APP JUDGE ALIGNMENT · p. 25

F DETAILED EVALUATION RESULTS

Model strengths vary across SDK subsystems and application domains. Claude-opus-5 leads application generation, while gpt-5.6-sol leads on design heuristics despite weaker functional application generation.

F DETAILED EVALUATION RESULTS

  • SDK strengths varied: gpt-5.6-sol led difficult gesture-input tasks, while gemini-3.8-flash led media streams and gemini-3.6-flash and gemini-3.7-flash excelled in spatial perception.gpt-5.6-sol and claude-opus-5 also showed strong depth and rendering completion; all models struggled with UI tasks requiring state management and translating raycasts into 2D updates.Source: F DETAILED EVALUATION RESULTS · p. 25, Table 8 · p. 26
  • Claude-opus-5 led application generation, topping education and health and tying for entertainment with deepseek-v4.1-flash and gemini-3.8-flash.Its health-task strength partly reflected its ability to render difficult visualizations.Source: F DETAILED EVALUATION RESULTS · p. 25, Table 9 · p. 26
  • Claude-opus-5's application-generation performance was partly attributed to its tendency to run runtime tests on applications.Table 11 reports development-tool usage, including runtime-test executions and test-to-edit ratios.Source: F DETAILED EVALUATION RESULTS · p. 25, Table 11 · p. 27
  • 9/10 heuristics: gpt-5.6-sol led design scores, producing more visually appealing, prompt-aligned applications, while claude-opus-5 was more functionally capable.deepseek-v4.1-flash, deepseek-v4-pro-0813, and claude-opus-5 tied for design-task performance.Source: F DETAILED EVALUATION RESULTS · p. 25, Table 10 · p. 27

G EXTENDED FAILURE ANALYSIS

This section documents XR application failures and successes, simulation limitations, actor/judge protocols, and evaluation prompts for spatial input responsiveness, reviewer interpretation, fixes, and evidence-based verification.

G EXTENDED FAILURE ANALYSIS

  • Failures are manually grouped into environment and perception, user tracking, and general development; category percentages are calculated over failed runs and can overlap.A single task may exhibit multiple failure modes; the supplied table captions identify pattern categories but do not provide their contents.Source: G EXTENDED FAILURE ANALYSIS · pp. 27–28

H EXAMPLES OF GENERATED APPS

  • Generated-app examples span healthcare, education, design, daily living, and entertainment, using capabilities from spatial interfaces and tracking to environmental sensing and spatial audio.Examples include virtual painting, cell and chemical visualization, volume measurement, games, and a virtual garden.Source: Figure 9 · p. 29
  • Examples show misplaced or noninteractive dots, virtual walls instead of cave paintings, hand-gesture and depth-tracking failures, and palm buttons embedded in users’ hands.In the cited tasks, only claude-opus-5 completed connect-the-dots; claude-opus-5 and deepseek-v4.1-flash implemented walking-clearance visualization successfully, while all shown palm-controller apps worked despite placement problems.Source: Figure 10 · p. 30, Figure 11 · p. 30, Figure 12 · p. 31

I LIMITATIONS

  • Simulator evaluations do not capture physiological or accessibility constraints, and XR Blocks supplies perfect sensor input unlike noisy, incomplete on-device sensing.The authors describe this as a best-case simulation condition and leave addressing these issues outside the work.Source: I LIMITATIONS · p. 32

J PROMPTS

  • The actor prompt requires one declared tool per turn, fresh observation-based checkpoints, and spatial evidence rather than visual overlap or reticle pixels to infer depth or interaction success.For ray selection it separates aiming from selection; held interactions require starting selection, moving while held, and ending selection with the same hand.Source: J PROMPTS · pp. 32–34
  • The trajectory judge prioritizes verifier probes and post-action observations, requires time-ordered evidence for transitions, and returns false for missing, ambiguous, stale, or contradictory evidence.Actions and actor exit messages show attempts or claims, not proof; spatial and interaction requirements need evidence of the relevant mode and resulting application response.Source: J PROMPTS · pp. 34–35
  • Evaluators assess whether 3D controls, interactive meshes, spatial buttons, and physics respond directly and naturally to spatial interaction.Source: J PROMPTS · p. 37
  • Virtual objects rendered over the physical environment should not be treated as an error.Source: J PROMPTS · p. 37
  • For each heuristic scored below 4, evaluators provide a concrete, step-by-step code or scene change to raise the score to 4 or 5.Source: J PROMPTS · p. 37
  • Evaluation notes should cite visible events and timestamps.Source: J PROMPTS · p. 37
Loading 2610.29014v1…