PaperlayerView exact revision on arXiv

Source-linked AI summary

XRFix: Exploring Performance Bug Repair of Extended Reality Applications with Large Language Models

Jingwen Wu, Hanyang Guo, Hong-Ning Dai, Xiapu Luo

arXiv:2608.21718v1cs.SE

TL;DR

XR performance bugs can degrade immersive experiences, yet XR-specific repair datasets, detectors, and repair tools are limited. XRFix builds an XR bug corpus and detection pipeline, then prompts five LLMs across three repair granularities; it achieves a 67.3% fix rate and outperforms three APR baselines. The framework’s scope is primarily Unity applications and ten typical bug types.

  • Problem

    XR performance repair lacks real-world datasets, accurate XR-specific detection tools, and effective repair frameworks, despite performance issues harming user experience.

  • Method

    XRFix combines 23 open-source XR projects, 104 real-world bugs, customized CodeQL and UnityLint analyzers, and prompts for three repair scenarios evaluated across five LLMs.

  • Results

    67.3% fix rate: XRFix outperforms the runner-up APR approach by 7.7% and achieves CodeBLEU of 0.664.

  • Takeaways & Limitations

    LLMs show potential for repairing XR performance bugs, and XRFix outperforms existing state-of-the-art APR approaches.

  • Takeaways & Limitations

    XRFix mainly covers ten typical bug types in Unity applications and lacks repair capabilities for visual and non-visual XR-function bugs.

Abstract

from arXiv · show

As an emerging technology, Extended Reality provides end-users with an immersive experience of interacting with virtual and physical environments. Unlike traditional software, the execution of XR applications involves more computationally complex operations, such as 3D scene rendering, real-time animation, and process simulations. Inefficient coding practices during the software development of XR applications may cause various performance bugs, degrading user experience and even causing motion sickness. Thus, it is an urgent need to develop an automated program repair framework for fixing performance bugs in complex XR programs. However, it is non-trivial to achieve this goal due to several technical challenges: (1) a lack of a real-world XR codebase and bug dataset, (2) no accurate bug detection tool, and (3) no effective bug-fixing tool designed for XR performance bugs. To tackle these challenges, we present a novel large language model-based framework, namely XRFix, to repair performance bugs for open-source XR programs. We first construct a corpus of domain-specific performance bugs built with a codebase from 23 open-source XR projects and a dataset of XR-related bugs containing 104 real-world bugs. Then, we tailor two static analysis tools for accurately detecting bugs in both C# scripts and asset files. Last, we design different prompts to instruct LLMs to fix XR bugs in three types of bug scenarios with different complexities, i.e., single-line level, function level, and class level. We conduct extensive experiments on five off-the-shelf LLMs to evaluate the bug-fixing performance of XRFix. We also compare our XRFix with three SOTA APR approaches. Through static analysis, reference answer comparison, and manual inspection, we demonstrate that our XRFix can effectively fix XR bugs, outperforming SOTA APR methods.

1 Introduction

XR applications demand substantial computation for rendering, animation, and simulation, making performance bugs especially harmful to user experience. XRFix addresses missing XR repair resources, detection tools, and tailored repair methods with an LLM-based framework.

  • Motivation: XR apps consume more GPU, CPU, and memory than conventional mobile apps, causing lagging, degradation, and frame discontinuity that can trigger motion sickness.These effects arise from complex operations such as 3D rendering, real-time animation, and process simulation on resource-constrained devices.
  • Research gap: Research has studied XR performance optimization and bug detection, but few efforts automate performance-bug repair for XR applications.
  • Challenges: XRFix targets three challenges: missing real-world XR bug data, insufficient XR-specific static detection, and ineffective repair frameworks customized for XR programs.
  • Approach: The framework collects 23 open-source Unity XR projects, summarizes ten bug types, and constructs a real-world XR bug corpus.
  • Approach: XRFix combines customized CodeQL and UnityLint analyzers with prompts for single-line, function-level, and class-level repair scenarios.
  • Evaluation: Experiments using static analysis, reference-answer comparison, and manual inspection show that XRFix outperforms existing state-of-the-art APR approaches.

2 Background

The background describes Unity’s lifecycle and its performance-related bug patterns, then introduces static analysis and LLM-based APR techniques relevant to XRFix.

  • 2.1 XR Unity-related Bugs: Unity XR scenes contain game objects controlled by C# scripts, with lifecycle stages spanning initialization, updates, input, logic, rendering, and decommissioning.
  • 2.1 XR Unity-related Bugs: Garbage collection frees unused memory but is unpredictable and costly; inappropriate resource allocation can accelerate it and affect animation.
  • 2.1 XR Unity-related Bugs: The paper considers ten Unity lifecycle bugs distributed across Physics Update, Logic Update, and Rendering, including simulation, rendering, and allocation problems.
  • 2.2 Static Code Analysis: Static analysis supports source-code inspection without execution, and XRFix uses Tree-sitter to extract C# functions and identify bug-pattern structures.
  • 2.2 Static Code Analysis: XRFix integrates UnityLint for scripts and asset files with CodeQL for complex source-code data flows and customized Unity-related queries.
  • 2.3 LLM-based APR: LLM-based APR commonly uses fine-tuning, few-shot, or zero-shot learning, and this study investigates prompt engineering with five state-of-the-art LLMs.

3 Methodology

XRFix builds an XR repair dataset from open-source repositories, detects Unity-related performance bugs with customized static analysis, and prompts LLMs to repair three bug-complexity levels. The resulting dataset contains 104 bugs drawn from a 23-repository XR codebase.

  • Workflow: XRFix’s workflow collects XR repositories, customizes static analyzers, generates LLM repair prompts, and evaluates incorporated fixes.Evaluation combines static analysis, reference-answer comparison, and manual inspection.
  • Data Collection: The collected sources include SideQuest and GitHub projects, with repository filtering based on XR keywords and Unity development.The initial collection includes 25 popular SideQuest projects and roughly 72 Unity-developed GitHub projects before further exclusions.
  • Data Collection: Researchers search repositories, forums, papers, and MITRE CWE using XR and performance keywords, then review candidates to identify 10 typical XR bugs.Two experienced XR researchers review the candidate results and assess whether bugs are suitable for LLM-based repair without human intervention.
  • Bug Detection: The final codebase contains 23 open-source XR repositories, using Tree-sitter, CodeQL, and UnityLint for code and asset analysis.Tree-sitter supports syntax-based extraction, CodeQL provides data-flow analysis, and UnityLint detects asset-file smells.
  • Repair Dataset: The dataset curates 104 bugs across direct, cross-function, and cross-class call scenarios, each paired with one or more author-provided reference answers.These scenarios organize increasing function-call complexity for program repair evaluation.
  • Prompt Design: Four prompt templates target single-line, function-level, and class-level repairs by supplying code contexts appropriate to each scenario.Class-level repairs may generate new functions from another class, while function-level repairs modify one or more functions within the same class.

4 Experiment

The experiment generates patches from multiple LLMs under context-window constraints, merges responses into original files, and evaluates them with static analysis, compilation checks, similarity measures, and manual inspection.

  • Patch Generation: XRFix queries GPT-3.5-Turbo, GPT-4o, Code Llama, StarChat-𝛽, and Deepseek-Coder to generate candidate patches.The first two models use the official ChatGPT API, while the other models are loaded through HuggingFace.
  • Prompt Construction: Prompt reduction retains using directives, the buggy function’s beginning, and template-specific code lines to address model context-window limits.The approach also fixes the maximum token budget for fair comparison.
  • Patch Merging: Tree-sitter merges generated functions, variables, or code lines into the original buggy file by comparing extracted AST features.Bug-free functions can be replaced when updated, and generated buggy functions or lines are inserted into the original code.
  • Evaluation: Evaluation checks bug removal with static analyzers, compilability through CodeQL database existence for C# scripts, reference-answer similarity, and manual inspection.When multiple reference answers exist, the highest similarity score is selected for comparison.
  • Metrics: Fix Rate measures bugs fixed, while Percentage of Plausible Fix measures plausible LLM responses.Average CodeBLEU and similarity scores are reported with bootstrap confidence intervals, and mean differences are tested with the Wilcoxon signed-rank test at α=0.05.

4.3 Compared Techniques

Because no existing APR method specifically targets XR performance bugs, XRFix is compared with three general C# APR approaches adapted to the XR dataset and evaluated across multiple candidate responses.

  • Compared Techniques: The comparison uses three state-of-the-art APR approaches for general C# bugs because no existing method is designed specifically for XR performance bugs.These methods were mainly evaluated previously on Defects4J and QuixBugs, so the authors adapt them to the XR dataset.
  • Evaluation Setup: Each baseline generates five responses per bug scenario for comparison with XRFix.The evaluation uses the candidate patches produced under the adapted baseline setup.
  • Baselines: CodeT5 is included as a model fine-tuned for multi-hunk APR tasks using C/C# code from GitHub.Its training on the CPatMiner dataset motivates its use for the comparison.
  • Baselines: The compared repair setup supplies buggy context with bug lines masked and evaluates top-5 candidate patches.Self-Repair uses test-failure feedback and brief failure explanations before generating repair information.
  • Research Questions: The experiments address baseline comparison through RQ6, which asks how XRFix’s repairing ability compares with baseline APR.The broader research questions also examine LLM performance, prompt engineering, and bug-fixing scenarios.

4.5 RQ1: Distribution of XR Program Repair Evaluation Dataset

The curated XR repair dataset contains 104 real-world bugs spanning open-source projects and three code-scenario levels. Evaluation results show varying LLM repair performance, with GPT-4o generally strongest and RHA particularly challenging.

  • Dataset distribution: 104 real-world bugs were curated from open-source XR projects with diverse bug sources and code scenarios.The dataset includes bug locations, names, descriptions, and reference answers.
  • Dataset distribution: 76 single-line bugs represent more than 73.07% of scenarios, compared with 22% function-level and 6% class-level bugs.IDU and RWT span all three scenarios, while other bug types cover fewer scenarios.
  • LLM repair performance: GPT-4o generated 70 plausible fixes under prompt c, achieving a 67.31% fix rate, slightly above GPT-3.5-Turbo’s 63.46%.Code Llama reached 45.19% with prompt c, Deepseek-Coder 42.31% with prompt e, and StarChat-β 23.08%.
  • Repair challenges: RHA was the most challenging bug type, with a 52.17% fix rate, because repairs require removing or commenting out buggy statements.Manual inspection found GPT-4o produced 8.33% more correct patches than GPT-3.5-Turbo, while Deepseek-Coder performed best among code LLMs.
  • LLM repair performance: GPT-4o achieved a 45% plausible-fix rate with prompt c, exceeding GPT-3.5-Turbo’s 30.77%.Code Llama and Deepseek-Coder reached 15.38% and 17.88%, respectively, while StarChat-β reached 9.04%.
  • Reference comparison: GPT-4o and GPT-3.5-Turbo obtained overall C# CodeBLEU averages of 0.608 and 0.600, respectively.Among code LLMs, Code Llama scored 0.485, Deepseek-Coder 0.445, and StarChat-β 0.393; GPT-4o achieved an asset-file average of 1.0 for SORL and AOBL.

4.8 RQ4: Evaluation on Prompt Templates in Program Repair.

Prompt templates affected repair outcomes across fix-rate, plausibility, and reference-similarity measures. Bug-specific instructions contributed most strongly, while the best prompt varied by model and file type.

  • 4.8.1 LLMs’ Performance with Different Prompt Templates.: Prompt c generally produced the best fix rates and plausible-response percentages across LLMs.The prompt comparison evaluates five templates across different models and repair outcomes.
  • 4.8.1 LLMs’ Performance with Different Prompt Templates.: GPT-4o achieved its highest C# CodeBLEU with prompt d at 0.700, while Code Llama and Deepseek-Coder performed best with prompt e.StarChat-β performed best with prompt c, and GPT-3.5-Turbo showed no significant difference between prompts c and d.
  • 4.8.1 LLMs’ Performance with Different Prompt Templates.: For asset files, GPT-4o, Code Llama, and Deepseek-Coder achieved their highest similarity scores with prompt c.The prompt yielding the best similarity therefore differed across file types and models.
  • 4.8.2 Ablation Study.: The ablation study measured Fix Rate, Percentage of Plausible Fix, and CodeBLEU against Basic Instruction as the baseline.Mean scores and standard deviations were calculated across five LLMs.
  • 4.8.2 Ablation Study.: Bug Instruction contributed the highest improvement across all three evaluation metrics.Fix Instruction also improved performance, but less than Bug Instruction.
  • 4.8.2 Ablation Study.: Table 9 compares plausible responses generated by LLMs under prompt c across single-line, function-level, and class-level scenarios.The comparison isolates bug-scenario complexity while holding the prompt variant constant.

4.9 RQ5: Performance of LLMs in different bug scenarios.

LLM performance differed across single-line, function-level, and class-level XR bugs. GPT-4o led all three scenarios under prompt c.

  • Bug-scenario performance: GPT-4o generated 199 of 380 plausible patches for single-line bugs under prompt c.It also ranked first for function-level and class-level scenarios.
  • Bug-scenario performance: GPT-4o generated 25 of 110 plausible fixes for function-level bugs and 10 of 30 for class-level bugs.GPT-3.5-Turbo ranked second for single-line and function-level bugs, while Code Llama ranked second for class-level bugs.
  • Bug-scenario performance: GPT-4o outperformed the other evaluated LLMs across all three bug scenarios.The results demonstrate model-dependent performance differences across scenario complexities.

4.10 RQ6: XRFix’s Repair Ability Compared with SOTA APR Methods.

XRFix, using GPT-4o with prompt c, outperformed three adapted state-of-the-art APR baselines in XR bug repair. Its results combined higher repair success with strong code similarity.

  • APR comparison: XRFix achieved a 67.3% fix rate, exceeding the runner-up APR approach by 7.7%.AlphaRepair and fine-tuned CodeT5 each fixed fewer than 7% of bugs.
  • APR comparison: XRFix achieved the highest percentage of plausible fixes and a CodeBLEU score of 0.664.The reported CodeBLEU result indicates higher correctness without decreasing code similarity.
  • APR comparison: XRFix produced more correct and reliable fixes than AlphaRepair, fine-tuned CodeT5, and Self-Repair.The paper identifies difficulty handling diverse XR scenarios and reliably explaining coding errors as baseline weaknesses.

4.11 Case Study and Discussion

XRFix combines LLMs with customized static analysis to repair XR performance bugs, while its effectiveness varies by model and bug type and remains bounded by validity threats.

  • Case Study: GPT-4o is presented fixing an IDU bug by using CodeQL-localized buggy lines and a prompt describing the required repair.The procedure comments out buggy contents, adds repair guidance, and displays GPT-4o’s generated code.
  • Discussion: GPT-4o generally achieves the strongest effectiveness among evaluated LLMs, while model capabilities differ across bug types.GPT-3.5-Turbo is ineffective for class-level bugs, whereas Code Llama achieves the highest fix rate and Deepseek-Coder performs consistently.
  • Threats to Internal Validity: XRFix’s performance depends on UnityLint and CodeQL, whose rule correctness was assessed alongside LLM fixes through static analysis, reference comparison, and manual inspection.The authors report high precision for the customized and predefined static-analysis rules.
  • Threats to External Validity: XRFix mainly covers ten typical XR bug types and Unity applications, which may limit scalability to other bugs and development frameworks.The paper names Unreal Engine and SteamVR as examples of other frameworks.
  • Threats to External Validity: Most collected bugs predate October 2023, creating a possible data-leakage threat; a nine-bug post-cutoff check produced consistent trends and fixes.The authors use the post-cutoff subset as evidence that conclusions still hold with low data leakage.
  • Threats to Construct Validity: XRFix currently lacks repair capabilities for visual and non-visual XR functions, including stereoscopic visual inconsistency bugs.The authors plan to extend the framework with tools such as large multimodality models.

5 Related Work

Prior XR research has studied development practices, bug characterization, and detection, while automated program repair has evolved through search-, constraint-, template-, and LLM-based methods. Pretrained code language models provide the broader modeling foundation.

  • XR Development Bugs: XR-related studies have examined development risks, WebXR bug scenarios, and performance phenomena, but much existing work overlooks bad practices or bugs in XR development.The cited studies include user and developer interviews and GitHub-based WebXR bug categorization.
  • Automated Program Repair: Automated program repair progressed from search-based and constraint-based techniques to template-based approaches and recent LLM-based zero-shot, few-shot, and self-repair methods.Examples include Nopol, SemFix, AlphaRepair, TypeFix, and iterative self-repair with feedback.
  • Pre-trained Language Models for Code: Pretrained code language models are commonly grouped into encoder-only, decoder-only, and encoder-decoder architectures with different pretraining objectives and code-task applications.CodeBERT and GraphCodeBERT are described as encoder-only models mainly used for code understanding tasks such as code search.

6 Conclusion

XRFix is an LLM-based framework for repairing performance bugs in open-source XR programs. Experiments across five LLMs and comparisons with three SOTA APR approaches indicate strong repair potential even with zero-shot learning.

  • Conclusion: XRFix constructs XR bug and project datasets, customizes static-analysis rules for bug localization, and uses prompt templates to guide LLM-based repair.The framework is evaluated across five state-of-the-art LLMs and compared with three SOTA APR approaches.
  • Conclusion: The experiments show that LLMs can effectively repair XR bugs with zero-shot learning, while future work targets multimodality and agentic intelligence for code understanding and localization.These future directions are intended to enhance XRFix for XR projects.
Loading 2608.21718v1…