PaperlayerView exact revision on arXiv

Source-linked AI summary

ExploreAI: Agentic Exploration Knowledge Bases for Reproducible Observable-Regression Testing of Black-Box VR and 3D Applications

Jiajie Wang, Kebin Peng, Wei Wang, Xiaoyin Wang, Sen He, Xue Qin

arXiv:2608.21628v1cs.SEcs.RO

TL;DR

Black-box VR/3D regression testing needs reproducible evidence because failures depend on spatial movement, visibility, and captured views. ExploreAI uses an LLM for exploration decisions while specialized modules execute perception, navigation, capture, and logging, producing per-object Exploration Knowledge Bases. Across six scenes, it achieves high-completeness exploration, and pilots show EKB-guided traces improve reproduction by humans and LLM-based agents.

  • Problem

    Black-box VR/3D regression evidence is difficult to reproduce because exploration runs differ in paths, visible objects, and captured views.

  • Method

    ExploreAI constructs a per-object Exploration Knowledge Base that links scan evidence, target selection, navigation paths, multi-view captures, and verification for reproducible testing.

  • Results

    Across six scenes, ExploreAI reaches 0.979 object completeness and 0.971 view completeness in complete exploration, while EKB-guided traces improve human and automated reproduction.

  • Takeaways & Limitations

    EKBs preserve how object-view evidence was found, captured, verified, and compared, supporting reproducible observable-regression testing across application versions.

  • Takeaways & Limitations

    The evaluation covers six scenes, and the reproduction studies are pilots rather than statistically powered human or agent studies.

Abstract

from arXiv · show

Black-box VR and 3D applications are difficult to regression test because observable failures depend on where a tester moves, what objects are visible, and which views are captured. Manual exploratory testing can find such failures, but its evidence is time-consuming to reproduce; systematic sweeps are reproducible, but they lack semantic guidance and spend exploration budget on low-value viewpoints. We observe that an LLM can make the high-level decisions a human tester makes during exploration: interpreting a task, choosing which objects to inspect, grouping related objects, recording what it saw, and deciding when missing evidence should trigger another attempt. Based on this observation, we present ExploreAI, an LLM-driven agentic framework that offloads repeated perception, navigation, multi-view capture execution, and logging to specialized modules while using the LLM for planning, evidence recording, capture-policy decisions, and verification decisions. ExploreAI constructs an Exploration Knowledge Base (EKB): a structured, per-object record of one exploration run. For each object the agent finds, the EKB stores the scan evidence that exposed it, the selected target, the navigation path, the multi-view capture, and the self-verification result. The EKB is a reusable testing artifact that supports reproducible observable-regression checking across versions of a VR or 3D application. Across six indoor and outdoor scenes in Unity, AI2-THOR, and BeamNG, ExploreAI constructs high-completeness EKBs under both complete and target exploration, and an LLM-module ablation shows where semantic planning, capture policy, evidence recording, and self-verification contribute. Reproduction pilots further show that EKB-guided traces help both humans and LLM-based reproducers reproduce exact object-view evidence more effectively than conditions without EKB context.

I. INTRODUCTION

EXPLOREAI frames black-box VR/3D regression testing as a reproducibility problem and constructs reusable object-centric evidence records. Its LLM-guided planning and specialized execution modules achieve high-completeness exploration and improve reproduction of object-view evidence.

  • Motivation: Black-box VR/3D testing is difficult because evidence depends on navigation, visible objects, viewing conditions, and captured views.Manual exploration can uncover failures, but its paths and screenshots are difficult to replay or compare.
  • Evaluation: Across six scenes, EXPLOREAI reaches 0.979 object completeness and 0.971 view completeness in complete exploration and completes all query-relevant targets in target exploration.These results are reported under equal scene budgets and against a frozen manifest.
  • Evaluation: The LLM-module ablation identifies contributions from evidence recording, planning, and self-verification, while EKB context improves human and automated reproduction of object-view evidence.Removing planning reduces Urban Cabin completeness, and reproduction benefits also extend to changed scene versions.
  • Approach: EXPLOREAI constructs an Exploration Knowledge Base (EKB) that preserves per-object traces, multi-view observations, navigation paths, and failure information for cross-version replay and comparison.The EKB is designed as a reusable testing artifact rather than a folder of screenshots.
  • Approach: The framework combines LLM-driven planning, target interpretation, capture planning, and completeness checking with specialized perception, navigation, capture, and logging modules.The LLM guides high-level decisions while repeated low-level execution is delegated to specialized modules.

II. RELATED WORK AND PROBLEM STATEMENT

Prior VR testing and environment-understanding systems support exploration, interaction, and semantic representation but do not preserve discovered evidence as a reusable regression artifact. ExploreAI addresses this gap by defining observable regression around reproducible object-view evidence from black-box applications.

  • Related Work: Prior VR/XR testing systems explore scenes, execute interactions, or generate scenarios, but they do not preserve discovered evidence as a reusable regression artifact.Related work focuses on testing effectiveness, navigation, or semantic reasoning rather than evidence preservation.
  • Problem Statement: The black-box setting restricts the tester to rendered frames and input controls without access to the scene graph, source code, or internal state.Correctness is judged from observable object-view evidence while moving through the application.
  • Problem Statement: An EKB stores scan evidence, selected targets, navigation paths, multi-view captures, and self-verification results for each object found during an exploration run.It also stores metadata, model logs, and failure cases.
  • Problem Statement: Observable regression is a version difference that changes what a tester can see, such as a removed object, swapped asset, blocked path, or lost view.The intended artifact supports reproducing the same object-view evidence and diffing later versions against it.

III. METHODOLOGY

ExploreAI combines LLM-guided semantic decisions with specialized perception and deterministic control to turn a black-box VR/3D testing task into a reproducible EKB. Its workflow covers exploration, target selection, navigation, multi-view capture, evidence recording, and verification.

  • Overview: ExploreAI converts a testing goal and running black-box VR/3D application into an EKB linking object evidence, navigation, views, recovery, and failures.The system supports complete exploration and category-focused target exploration.
  • Perception and Planning: A seed sweep collects rendered frames and detections before the LLM interprets the task, prioritizes targets, and records semantic evidence and decisions.The planner maps task text to detectable categories, tracks scene risks, and decides which objects or groups to inspect.
  • Execution Workflow: The workflow executes selected targets through far- and near-view navigation, LLM-driven multi-view capture, EKB updates, and self-verification.Fig. 2 distinguishes LLM decision points from repeated low-level execution by perception and deterministic control.
  • Perception and Planning: YOLO-backed perception handles frequent visual confirmation, while the LLM is reserved for semantic decisions and evidence recording.This division avoids using costly LLM calls for every field-of-view judgment, movement step, or low-level confirmation.
  • Perception and Planning: Candidate ranking combines semantic relevance, visual support, prominence, distance, diversity, and repeat penalties to allocate exploration budget.Weights are fixed in each run configuration, with broader relevance and diversity in complete exploration and query matching in target exploration.

C. Navigation: Far-View and Near-View Control

ExploreAI uses staged deterministic navigation to move from a candidate region to a centered, stable target view suitable for capture. Far-view movement approaches the target, while near-view control centers the detector box and enforces an orbiting distance band.

  • Far-View Navigation: Far-view navigation moves toward the candidate region while maintaining target direction stability.It stops at the configured approach radius, sufficient visual centering, or movement-budget exhaustion.
  • Far-View Navigation: YOLO confirms targets from ordinary rendered frames without simulator-specific object APIs or an LLM call at every control step.This supports fast repeated object confirmation during navigation.
  • Near-View Navigation: Near-view navigation uses detector bounding-box error to center and stabilize the camera before capture.The controller also checks that the target remains within the allowed distance band for orbiting.
  • Near-View Navigation: If a target disappears, becomes blocked, or cannot be centered within the step budget, the trace is marked as a path failure or target-lost failure.

D. LLM-Driven Multi-View Capture

After stabilization, ExploreAI captures object-centric evidence using an LLM-driven policy that adapts required views to object geometry and accessibility. The controller executes orbit paths while the LLM records intent, missing views, and recovery choices.

  • Capture Policy: The capture policy requires four views for free-standing objects and fewer reachable views for wall-attached or scene-boundary-blocked objects.The standard four-view denominator is front, left, back, and right.
  • Capture Execution: The controller follows an orbit path, while the LLM labels requested views and records the expected evidence.The LLM can group views for functional object sets and determine whether a wall-adjacent target needs two rather than four views.
  • Recovery: When one orbit direction is blocked, the capture policy can switch direction to attempt the missing side from the opposite route.

E. EKB Schema, Coverage, and LLM-Driven Self-Verification

The EKB represents each explored object as a trace containing discovery, semantic, navigation, view, recovery, and failure information. Coverage is evaluated against a frozen manifest, while failed traces remain evidence of incompleteness.

  • EKB Schema: An object trace records its first observation, supporting detections, assigned target category, approach, captured views, and self-verification outcome.
  • EKB Schema: Each trace tuple includes a stable identifier, semantic category, estimated position, detections, scan evidence, navigation path, views, recovery status, and failure record.The EKB also stores run configuration, task mode, LLM plan, detector model, videos, and per-target metrics.
  • Coverage: EKB coverage includes object, view, navigation-path, and log completeness.
  • Coverage: Completeness is measured against a frozen manifest rather than the system’s own plan.A failed trace remains in the EKB and contributes evidence for incompleteness instead of disappearing from the denominator.

IV. EXPERIMENT SETUPS

The evaluation isolates the value of the Exploration Knowledge Base and its LLM-guided decisions in reproducible black-box VR and 3D regression testing. Comparisons use matched frameworks with and without EKB context and module ablations.

  • Recovery and regression procedures: The missing-trace recovery procedure identifies incomplete traces, rescans from an alternate side, reranks candidates, and records unresolved evidence when recovery fails.Recovery uses scan evidence, wall policy, and failed paths, subject to the remaining exploration budget.
  • Recovery and regression procedures: The regression procedure matches baseline and candidate traces, reports missing objects or lost views, and ignores benign changes preserving category, reachability, and expected views.Stable identifiers are preferred; otherwise matching uses category, region, and supporting views.
  • Evaluation design: The study compares EXPLOREAI with and without EKB context to isolate the artifact’s contribution to reproducible testing evidence.The comparison addresses whether EKB context improves regression evidence reproduction rather than only exploration performance.
  • Evaluation design: Module ablations isolate the contributions of LLM-guided planning, capture policy, evidence recording, and verification decisions.The ablation targets individual decision modules while retaining the broader framework comparison.

A. Research Questions

The experiments ask whether EXPLOREAI produces complete, affordable EKBs, which modules contribute, and whether EKBs support reproducible cross-version regression testing. These questions are evaluated across six varied scenes with fixed, repeated budgets and frozen-manifest checks.

  • Research questions: RQ1 measures object, view, navigation-path, and log completeness, aggregate completeness across six models and scenes, and token and API cost.Complete and target exploration are evaluated under the same protocol.
  • Research questions: RQ2 uses module ablations over six scenes, with metrics aligned to the four EKB completeness components and aggregate completeness.The design focuses on the contribution of individual, especially LLM-driven, modules.
  • Research questions: RQ3 evaluates human and automated reproduction with and without EKB context and checks regression detection using injected object, asset, path, and view changes.Reproduction metrics include object completeness, view completeness, time, and token use.
  • Experimental setting: The benchmark covers three indoor and three outdoor scenes spanning dense furniture, room-scale navigation, roads, low-poly environments, and vehicle-oriented settings.The scenes use Unity, AI2-THOR, and BeamNG.
  • Ground truth: Completeness is computed against a frozen manifest independently checked by two annotator passes before metric computation.Final agreement was Cohen’s kappa = 0.86 and object-id Jaccard agreement = 0.89 before adjudication.
  • Experimental setting: Each model-scene pair uses ten repetitions, with five complete-exploration and five target-exploration runs under fixed scene-level budgets.Fixed budgets preserve comparability across exploration modes and LLM-module ablations.

V. EVALUATION

Across six scenes, EXPLOREAI constructs highly complete EKBs in complete exploration and completes all query-relevant targets in target exploration. Completeness saturates across tested model configurations, while residual failures mainly reflect dynamic visibility and geometric constraints.

  • RQ1 results: 0.979 object completeness and 0.971 view completeness are achieved as macro averages in complete exploration, while target exploration completes all query-relevant targets.Completeness is measured against frozen scene manifests for complete exploration and query-relevant target subsets for target exploration.
  • Model generality: All six LLM configurations reach the same high aggregate EKB completeness across six scenes and both exploration modes.The benchmark’s deterministic perception, navigation, and reachability layers cap completion on these scenes.
  • Model generality: SD = 0 completeness variance is observed across six model families and five repetitions for each scene-mode cell.The residual Urban Cabin gap, 0.871/0.825, is attributed to a fixed geometric-visibility limit rather than model capability.
  • Cost: Table IV reports mean tokens per run, cost per run, and time per run separately for complete and target exploration across six LLM configurations.The protocol uses ten repetitions per model, split evenly between the two exploration modes.
  • Failure analysis: Residual failures are mostly dynamic-visibility failures, including objects near room boundaries or partially blocked by furniture.The controller records missing views instead of silently counting affected traces as complete.

B. RQ2: How much do the individual modules, especially LLM-driven decisions, contribute?

The ablation isolates how EXPLOREAI’s LLM decisions affect EKB completeness and shows that different hooks govern semantic targeting, navigation context, and multi-view evidence. Reproduction experiments further indicate that EKB context improves evidence reproduction for humans and automated agents.

  • LLM decision ablation: Removing LLM planning causes the largest semantic drop, with detector-only exploration repeatedly selecting nearby furniture and missing a partially occluded shelf.The ablation replays the same logged perception and navigation traces while disabling one LLM hook at a time.
  • LLM decision ablation: Removing LLM evidence recording preserves object and view detection but reduces navigation-path and log completeness.The EKB also loses the semantic rationale and reproducible navigation context for the recorded exploration.
  • LLM decision ablation: Removing multi-view capture or self-verification mostly reduces view completeness because the system no longer reasons about reachable sides or incomplete traces.
  • Reproduction: EKB context improves both human and automated reproduction, especially when the scene version contains changed elements.The reproducer must still navigate and capture the evidence; EKB context supplies prior traces rather than the target evidence itself.
  • Cross-version detection: DiffEKBs detect 35 of 36 injected cross-version changes with no false positives, while the only miss is a wrong-object case.The authors characterize this as a controlled-injection proof of concept, with real version-to-version detection left for future work.

VI. THREATS TO VALIDITY

The validity discussion identifies measurement, provenance, scope, and study-design constraints on interpreting completeness, ablation, reproduction, and regression-detection results. The evaluation measures observable object-view evidence rather than general behavioral correctness and does not establish performance across all VR systems.

  • Internal validity: Object and view completeness can contain measurement error because detectors may miss partially visible targets, assign wrong labels, or capture incomplete angles.The study mitigates this by preserving scan evidence, saved views, detector confirmations, failure cases, and video context.
  • Internal validity: The studies use different denominators, so planned-target completion must not be treated as scene-level completeness.The authors report the complete-flow matrix, generality benchmark, ablation, and reproduction study separately.
  • Construct validity: The metrics judge observable object-view evidence, not code coverage, branch coverage, or general behavioral correctness.Claims about general behavioral correctness require additional monitors or instrumentation.
  • External validity: The evaluation covers six scenes across Unity, AI2-THOR, and BeamNG, but these scenes do not represent all VR applications.The scene-level RQ1 sample is benchmark evidence and effect-size estimation rather than definitive statistical separation of all testing methods.
  • External validity: RQ2 uses trace-replay ablations instead of full end-to-end reruns for every removed LLM hook.This isolates named decisions, while the authors identify end-to-end reruns in every simulator as future work.
  • Study scope: The six-participant human reproduction study and automated reproduction study are pilots rather than statistically powered studies.Larger human and programmatic studies are needed to estimate effects across testers, models, and applications.

VII. CONCLUSION

ExploreAI constructs Exploration Knowledge Bases that support reproducible black-box VR/3D regression testing by preserving both exploratory decisions and object-view evidence. Across six scenes, its EKB traces support exact evidence reproduction by humans and LLM-based agents.

  • ExploreAI constructs EKBs for reproducible black-box VR/3D regression testing.
  • The framework uses an LLM for exploratory-testing decisions while specialized modules handle perception, navigation, multi-view capture, and logging.
  • Across six scenes, ExploreAI produces high-completeness object-view evidence under both complete and target exploration.
  • Reproduction pilots show that EKB traces help humans and LLM-based agents reproduce exact object-view evidence.
Loading 2608.21628v1…