PaperlayerView exact revision on arXiv

Source-linked AI summary

Grounded GUI Understanding for Vision-Based Spatial Intelligent Agent: Exemplified by Extended Reality Apps

Shuqing Li, Binchang Li, Yepang Liu, Cuiyun Gao, Jianping Zhang, Shing-Chi Cheung, Michael R. Lyu

arXiv:2409.10811v4cs.SEcs.AIcs.CVcs.HCcs.MM

TL;DR

XR IGE detection is challenging because XR scenes contain heterogeneous open-vocabulary elements, context-dependent interactability, and demanding localization requirements. Orienter uses semantic context before detection and iteratively validates candidates; it outperforms state-of-the-art GUI detection approaches and improves automated testing results.

  • Problem

    2D IGE detection approaches rely on large labeled datasets and predefined categories, while XR presents open-vocabulary elements, context-sensitive interactability, and spatial-localization challenges.

    Source: §1 · pp. 2–3

  • Method

    Orienter is a zero-shot XR IGE detection framework that comprehends semantic context before detection, iteratively validates candidates, and classifies interactability using that context.

    Source: §1 · p. 3

  • Results

    Orienter outperforms the listed baselines on almost all interactability and semantics metrics, with F1 improvements up to 3.7× and 121.4×, respectively.

    Source: §1 · p. 4

  • Takeaways & Limitations

    Orienter-guided testing covers 103.1% more IGEs and achieves 125.7% more effective interactions than testing without action-space isolation.

    Source: §1 · p. 4

    Effectiveness demonstrated in specific scenarios may not generalize uniformly to XR environments with unique interaction paradigms or novel GUI element types.

    Source: §7 · p. 19

Abstract

from arXiv · show

In recent years, spatial computing a.k.a. Extended Reality (XR) has emerged as a transformative technology, offering users immersive and interactive experiences across diversified virtual environments. Users can interact with XR apps through interactable GUI elements (IGEs) on the stereoscopic three-dimensional (3D) graphical user interface (GUI). The accurate recognition of these IGEs is instrumental, serving as the foundation of many software engineering tasks, including automated testing and effective GUI search. The most recent IGE detection approaches for 2D mobile apps typically train a supervised object detection model based on a large-scale manually-labeled GUI dataset, usually with a pre-defined set of clickable GUI element categories like buttons and spinners. Such approaches can hardly be applied to IGE detection in XR apps, due to a multitude of challenges including complexities posed by open-vocabulary and heterogeneous IGE categories, intricacies of context-sensitive interactability, and the necessities of precise spatial perception and visual-semantic alignment for accurate IGE detection results. Thus, it is necessary to embark on the IGE research tailored to XR apps. In this paper, we propose the first zero-shot cOntext-sensitive inteRactable GUI ElemeNT dEtection framework for virtual Reality apps, named Orienter. By imitating human behaviors, Orienter observes and understands the semantic contexts of XR app scenes first, before performing the detection. The detection process is iterated within a feedback-directed validation and reflection loop. Specifically, Orienter contains three components, including (1) Semantic context comprehension, (2) Reflection-directed IGE candidate detection, and (3) Context-sensitive interactability classification. Extensive experiments demonstrate that Orienter is more effective than the state-of-the-art GUI element detection approaches.

1 Introduction

XR IGE detection matters for software engineering, but approaches built for fixed 2D categories face open-vocabulary elements, context-dependent interactability, and spatial-localization challenges. Orienter uses context comprehension and feedback-directed detection; the paper also introduces a benchmark and reports stronger detection and testing results.

1 Introduction

  • Accurate IGE recognition supports software-engineering tasks including automated testing and GUI code generation.Source: §1 · p. 2
  • XR apps contain open-vocabulary, heterogeneous IGE categories, while an element’s interactability can depend on app and scenario context rather than appearance alone.Elements may also be occluded or viewed from non-front perspectives, increasing visual variation.Source: §1 · pp. 2–3
  • Precise spatial perception and visual-semantic alignment are necessary for accurate XR IGE localization, yet LMMs can produce imprecise or misaligned bounding boxes.Source: §1 · p. 3
  • Orienter comprehends semantic context before detection and iteratively validates and refines IGE candidates through feedback-directed reflection.Its three components are context comprehension, reflection-directed candidate detection, and context-sensitive interactability classification.Source: §1 · p. 3
  • The paper constructs a benchmark dataset for evaluating XR IGE detection.Source: §1 · p. 3
  • Orienter outperforms the listed baselines on almost all interactability and semantics metrics, with F1 improvements up to 3.7× and 121.4×, respectively.Orienter-guided testing also covers 103.1% more IGEs and achieves 125.7% more effective interactions than testing without action-space isolation.Source: §1 · p. 4

2 Background on Extended Reality (XR)

XR users interact with GUI elements through varied devices and actions, within immersive scenes that may combine virtual and physical content. Motivating examples show that an LMM can misidentify elements and localize them poorly, motivating added context and techniques.

2 Background on Extended Reality (XR)

  • XR systems span VR, AR, and MR, with devices including head-mounted displays, controllers, gesture-recognition systems, and tracking systems.Source: §2 · p. 4
  • Users can interact with XR GUI elements by pressing buttons, touching, gazing, or using gestures and movements.Source: §2 · pp. 4–5

2.1 Motivating Examples

  • In a motivating study, two authors sampled 30 Steam VR apps and prompted GPT-4o to identify and locate IGEs in screenshots.Source: §2.1 · p. 5
  • The examples show an LMM mistaking a mailbox for a “Missile-like Object” and producing boxes that partially cover objects, mismatch labels, or mark backgrounds.Source: §2.1 · p. 5
  • The authors conclude that LMMs alone cannot effectively detect IGEs in complex XR apps and propose adding context and integrating other techniques.Source: §2.1 · p. 5

3 The Orienter Approach

The paper formulates XR IGE detection using a single-eye 2D projection of a stereoscopic 3D scene. Orienter then comprehends scene context, detects and refines candidates, and classifies their interactability.

3.1 Problem Formulation of IGE Detection

  • A single-eye 2D projection can serve as the detection scene because the two eye views are projections of the 3D world and one can be recovered from the other.Source: §3.1 · p. 5
  • The detection formulation identifies each GUI element by interactability, spatial location, and semantic labels.The location is specified by a bounding box; semantic-label granularity depends on the XR scene.Source: §3.1 · pp. 5–6

3.2 Overview of Orienter

  • Orienter synthesizes global app context and local field-of-view context before detection, then mines element characteristics and iteratively validates and refines candidate IGEs.Source: §3.2 · p. 6
  • Orienter classifies detected candidates’ interactability using semantic context from the first stage and LMM chain-of-thought reasoning, without labeled training data.Source: §3.2 · p. 6

3.3 Module I: Semantic Context Comprehension

Orienter first establishes an XR app’s semantic context so subsequent detection can account for context-dependent interactability. It combines app-level global context with a visual summary of the current scene.

3.3 Module I: Semantic Context Comprehension

  • Together, global app context and local scenario context support context-sensitive analysis of XR scenes before IGE detection.The context is analyzed across both global and local granularity and text and vision modalities.Source: §3.3 · pp. 6–7

3.3.1 Global Context Extraction.

  • Orienter extracts global context from app-store details, including the app’s genre, themes, gameplay, interaction mechanisms, device support, and language.The attributes were identified through manual inspection of 30 VR applications.Source: §3.3.1 · p. 7

3.3.2 Local Context Perception.

  • It derives local context by asking an LMM to summarize the current XR scene from its screenshot, informed by the extracted global context.The summary covers GUI elements and background content in the user’s field of view.Source: §3.3.2 · p. 7

3.4 Module II: Reflection-Directed IGE Candidate Detection

Orienter uses semantic context to identify IGE candidates, describe them from multiple characteristics, and localize them with a language-aligned visual foundation model. A feedback loop checks detections and revisits uncertain candidates until an advisor reports sufficient confidence.

3.4 Module II: Reflection-Directed IGE Candidate Detection

  • Orienter mines multiple characteristics of each IGE candidate, formulates questions about them, and uses context-informed LMM reasoning to produce distinguishing descriptions.The characteristics include appearance, functionality, interaction mechanisms, and relative location.Source: §3.4 · pp. 8–9

3.4.2 Described IGE Candidate Detection.

  • VFMD aligns characteristic descriptions with XR-scene visual features to verify candidate existence and predict candidate locations.It combines language embeddings of descriptions with visual embeddings, then uses text-conditioned object queries and a visual-language alignment module.Source: §3.4 · p. 7, §3.4.2 · p. 9

3.4.3 Feedback-Directed Reflection.

  • An LMM advisor compares VFMD detections with mined characteristics, using mirrored regional checks for detections and reflection on missed descriptions before deciding whether to repeat mining.The loop ends when the advisor considers all detection results sufficiently confident.Source: §3.4 · p. 8, §3.4.3 · pp. 9–10

3.5 Module III: Context-Sensitive Interactability Classification

After establishing GUI elements and their locations, Orienter classifies which elements are interactable by reasoning within the app’s semantic context. The reasoning distinguishes interactive objects from visually similar background elements whose role depends on the app.

3.5 Module III: Context-Sensitive Interactability Classification

  • Orienter guides an LMM to reason in the app’s semantic context about whether each GUI element affords interaction, using examples to distinguish interactive objects from scenery.A tree may be interactable in a tree-planting game but background scenery in a fishing game.Source: §3.5 · p. 10

4 Dataset Construction

The authors construct an XR IGE dataset because they report no existing dataset for this task. Annotators collected and labeled app imagery, assigning context-sensitive IGE categories, and the resulting dataset spans diverse apps and semantics.

4 Dataset Construction

  • Thirteen annotators with at least two years of CS or electrical-engineering background spent more than three months collecting and annotating XR GUI images.They interacted with app elements, recorded their views, selected images, and received training and annotation guidelines.Source: §4 · pp. 10–11

4.1 Collection of XR Apps

  • The authors sampled 102 apps from 4,610 VR-only Steam apps to cover varied categories, contexts, and interaction paradigms.The sample spans 245 community-generated genres and was selected for a stated 95% confidence level and 10% margin of error.Source: §4.1 · p. 11

4.2 Collection of GUI Images from XR Apps

  • Annotators interacted with GUI components in selected apps, recorded stereo views, and saved images covering the explored scenes and elements.The dataset uses right-eye images from the stereo-view recordings.Source: §4.2 · p. 11

4.3 Annotation of IGEs in GUI Images

  • Annotators labeled IGE bounding boxes and context-dependent semantics, choosing category granularity to fit each app’s scenario.Fish can be one coarse category when individuals are semantically equivalent, but split into large and small fish when their roles differ.Source: §4.3 · p. 11, Figure 12 · p. 11

4.4 Dataset Statistics

  • The resulting dataset contains 1,552 images from 100 apps, with 4,470 IGE annotations across 766 semantic categories.It covers 12 official genres and 245 community-generated Steam categories; annotations use COCO-format bounding boxes.Source: §4.4 · p. 11

5 Experiment Design

The experiments assess Orienter’s detection performance, component contributions, and potential to improve automated XR testing. They compare it with direct-prompting, GUI-detection, and object-detection baselines using datasets, splits, and metrics tailored to these questions.

5.1 Research Questions

  • RQ1 evaluates interactability, semantics, and context-sensitive interactability; RQ2 examines the contribution of each Orienter component.The RQ3 comparison asks how Orienter can improve automated testing.Source: §5.1 · p. 12

5.2 Baselines

  • Baselines include direct prompting with GPT, Gemini, or Claude, alongside GUI parsers and trained object detectors.The listed methods include Xianyu, UIED, OmniParser, CenterNet2, Faster R-CNN, and YOLO v8.Source: §5.2 · p. 12, Figure 14 · p. 12

5.3.1 Implementation of Orienter.

  • Orienter uses four representative LMMs and APE-L_D for described IGE detection, then filters oversized boxes and applies NMS to reduce duplicates.NMS keeps the highest-confidence box among overlapping boxes with IoU above 0.7.Source: §5.3.1 · p. 12

5.3.2 Dataset Preparation.

  • Three dataset variants test semantic labels, binary interactability, and context-sensitive interactability across app, genre, and context-sensitive splits.Images are split approximately 6:1:3 into training, validation, and test sets; the context dataset includes 41 sampled categories and corresponding non-interactable objects.Source: §5.3.2 · pp. 12–13, Table 1 · p. 13

5.3.3 Experimental Setup.

  • Baselines are trained on training data and evaluated on test data; semantic matching uses embedding similarity above a threshold of 0.85.The evaluation uses a modified COCO API; context-sensitive predictions are evaluated on the context dataset after baseline training on the semantic dataset.Source: §5.3.3 · p. 13

5.3.4 XR App Automated Testing Setup.

  • RQ3 tests whether Orienter guidance improves black-box XR testing across 77 apps, using 60-minute runs and 30-run averages.The guided strategy samples Orienter-predicted boxes with probability p=1−t/T, while retaining random exploration of the full image.Source: §5.3.4 · pp. 13–14

5.4 Evaluation Metrics

  • Detection uses Precision, Recall, F1, and mAP, while testing uses IGE coverage and effective interaction count.An IGE is covered when an interaction point falls inside its bounding box; an interaction is effective when it falls inside any IGE box.Source: §5.4 · p. 14

6 Results and Analysis

Orienter generally outperforms the baselines on interactability, semantic detection, and context-sensitive interactability, while ablations show performance losses when components are removed. Orienter-guided testing improves overall results, though category-level outcomes vary.

6.1.1 RQ1-1: Performance in terms of interactability.

  • At IoU 0.75, Orienter with Gemini achieved 49.45% Precision versus Faster-RCNN at 33.42%, and 46.00% F1 versus CenterNet2 [63] at 29.32%.At IoU 0.95, Orienter Precision was 19.02%, over 321.7% above CenterNet2 [63].Source: §6.1.1 · p. 15, Table 2 · p. 15

6.1.2 RQ1-2: Performance in terms of semantics.

  • At IoU 0.75, Orienter with Gemini achieved 23.72% Precision on the genre split versus CenterNet2 [63] at 0.62%; at IoU 0.95 its F1 was 14.31% versus YOLO v8 at 0.40%.Orienter’s semantic mAP with Gemini declined from 22.12% at IoU 0.75 to 7.41% at 0.95.Source: §6.1.2 · pp. 15–16, Table 3 · p. 16

6.1.3 RQ1-3: Performance in terms of context-sensitive interactability.

  • At IoU 0.75, Orienter with GPT-4o achieved 32.51% F1 for context-sensitive interactability, surpassing CenterNet2 [63] by 6274.5%.At IoU 0.95, Orienter’s F1 remained above 10% while baseline metrics were approximately zero; Orienter with Gemini had 53.35% Precision.Source: §6.1.3 · p. 16, Table 3 · p. 16

6.2 RQ2: Ablation Study

  • Removing context comprehension reduced F1 by 14.10% for interactability and 11.07% for semantics at IoU 0.75; removing interactability classification caused the largest average F1 declines.Replacing IGE Feature Mining & Referring led to average F1 declines of 19.04% and 12.37% for interactability and semantics, respectively; removing interactability classification led to declines of 34.60% and 18.94%.Source: §6.2 · p. 17, Table 4 · p. 17

6.3 RQ3: Effectiveness for boosting automated testing

  • Orienter-guided testing improved coverage in most app categories, by up to 36.60% in Puzzle, but slightly underperformed random testing in Multiplayer.The Multiplayer coverage was around 0.03 for both strategies; category-wise evaluation covered 76 of 77 AUTs and omitted categories with fewer than 10 apps.Source: Table 5 · p. 17, Figure 16 · p. 18, §6.3 · p. 18
  • 0.83 IGE coverage and 13,038.03 effective interactions were reached by Orienter-guided testing, versus 0.65 and 5,911.53 under random testing.Guided testing had 0.64 coverage at 10 minutes versus 0.30 for random testing; the effective-interaction totals are averages across 936 GUI scenes.Source: Figure 15 · p. 18, §6.3 · p. 18

7 Threats to Validity

The authors identify annotation subjectivity and pretrained-model bias as internal validity threats, and limited generalizability across diverse XR apps as an external validity threat.

Internal Validity

  • Annotation judgments may vary across 13 annotators, and pretrained LMM biases could affect IGE predictions, especially in contexts underrepresented in training data.The authors describe training annotators for at least two hours and selecting models they consider representative as mitigations.Source: §7 · p. 19

External Validity

  • Effectiveness in the sampled apps may not generalize uniformly to XR environments with unique interaction paradigms or novel GUI element types.The authors sampled popular, complicated apps spanning many categories to mitigate this threat.Source: §7 · p. 19

8 Related Work

The related work covers IGE detection, XR testing, and exploratory studies of XR apps. It describes prior approaches and identifies open-vocabulary categories as a limitation of existing IGE detection methods.

8.1 IGE Detection

  • Prior IGE detection work includes traditional computer vision, deep learning, and combined approaches, with datasets and empirical studies focused largely on common, finite categories.The cited work includes mobile-game benchmarks and methods for GUI testing and widget localization.Source: §8.1 · p. 19

8.2 VR/AR Testing

  • XR testing research includes empirical studies and tools for code-test recommendations, test-data generation, AR test oracles, scene exploration, and visual inconsistency detection.The passage also describes an approach combining metamorphic testing, agent-based testing, and machine learning.Source: §8.2 · pp. 19–20

8.3 Exploratory Studies on VR/AR Apps

  • Exploratory XR studies examine project trends, bugs, user and developer concerns, performance optimization, software quality, and security and privacy.Source: §8.3 · p. 20

9 Conclusion

The conclusion presents Orienter as a zero-shot, context-sensitive framework for XR IGE detection and reports its effectiveness relative to state-of-the-art GUI detection approaches. It also reports benefits for automated GUI testing and introduces a dedicated XR IGE detection dataset.

9 Conclusion

  • Orienter is presented as the first zero-shot, context-sensitive IGE detection framework for XR apps.The paper also reports creating the first dedicated dataset for XR IGE detection.Source: §9 · p. 20
  • Experiments report that Orienter is more effective than state-of-the-art GUI element detection approaches.The conclusion also states that Orienter benefits automated GUI testing.Source: §9 · p. 20
  • The authors analyze key challenges in XR IGE detection and make the framework and dataset publicly available to spur further research.Source: §9 · p. 20

Related papers

Loading 2409.10811v4…