PaperlayerView exact revision on arXiv

Source-linked AI summary

Harnessing Vision-Language Models for Perceptual Quality Assessment and Autonomous Content Adjustment in Augmented Reality

Elias Rotondo, Lin Duan, Yanming Xiu, Sangjun Eom, Conrad Li, Maria Gorlatova

arXiv:2610.00677v1cs.CV

TL;DR

AR quality evaluation through user studies is costly and limits iterative design. The paper introduces RateAR, evaluates commercial VLMs, and builds an automated adjustment system; VLM predictions closely track human judgments, and participants report improved placement and size coherence.

  • Problem

    User studies are labor-intensive and time-consuming, bottlenecking iterative AR design and deployment.

    Source: §1 · p. 2

  • Method

    The paper introduces RateAR, evaluates eleven commercial VLMs on three AR visual factors, and builds a VLM-driven content-adjustment pipeline.

    Source: §1 · p. 2, §8 · p. 10

  • Results

    VLM predictions closely tracked human judgments, with Spearman’s rank-order correlation up to 0.8695, and over 90% of participants reported improved placement and size coherence.

    Source: §1 · p. 2, §8 · p. 10

  • Takeaways & Limitations

    VLM feedback can support automated, closed-loop correction of AR rendering, including placement and sizing.

    Source: §5.4.1 · p. 8

    Near-real-time adjustment remains outside the current scope because VLM inference is the primary latency bottleneck.

    Source: §7 · p. 10

Abstract

from arXiv · show

Advancements in augmented reality (AR) continue to foster innovative solutions, facilitating novel methodologies within educational systems, healthcare delivery, and risk-mitigation protocols. However, optimizing for end-user immersion and comfort remains challenging, as AR head-mounted displays contend with constrained scene geometry, spatial jitter, and temporal instability. User studies are the standard AR evaluation method for visual quality, but their cost, diminishing scalability, and inflexibility pose bottlenecks during iterative application design. To address this problem, we present an automated framework for AR content evaluation and refinement, built on vision-language models (VLMs), to evaluate and predict the visual fidelity of AR scenes as perceived by users. First, we introduce RateAR, a benchmark of AR images and videos collected across diverse scenes and environmental conditions, with good-to-excellent reliability (ICC(2,5) >= .90) across perceptual factors, including object placement, scale, and shadow consistency. Subsequently, we evaluate eleven commercial VLMs on the crafted benchmark. Results support that VLM-based quality predictions strongly correlate with human subjective judgments, achieving Spearman's rank-order correlations of up to 0.8695. An ablation study further suggests that, compared to other prompting strategies, our contextual prompting yields better alignment with human ratings while balancing introduced complexity cues. Building on these findings, we construct an automated AR content adjustment system and conduct a 21-participant user study. More than 90% of participants found that the system improved placement and size coherence of virtual content.

1 Introduction

The paper targets perceptual AR artifacts and the cost of evaluating them through user studies. It evaluates commercial VLMs and proposes an automated adjustment framework, reporting alignment with human ratings and user-perceived improvements.

1 Introduction

  • User studies are labor-intensive and time-consuming, bottlenecking iterative AR design, while existing automatic quality metrics need evaluation against human-centric AR principles.Inconsistent scene geometry can yield implausible shadows, incorrect scaling, and misaligned placement that disrupt immersion.Source: §1 · p. 2
  • The study asks whether commercial VLM ratings align with human judgments, whether a VLM-driven closed loop can refine AR content, and how prompts affect visual reasoning.The work introduces RateAR, a benchmark of 321 AR images and 112 videos, and evaluates eleven commercial VLMs on placement, size, and shadow quality.Source: §1 · p. 2
  • Spearman’s rank-order correlation reached 0.8695, and over 90% of participants reported improved AR content placement and size coherence.The authors also report that GPT-model ratings numerically outperformed the quality-assessment baselines.Source: §1 · p. 2

2 Related Work

Prior AR perception research relies heavily on controlled human judgments, while automatic evaluators can be narrow or insufficiently robust. Existing VLM work in AR emphasizes generation and placement more than factor-specific perceptual quality assessment.

2.1 AR Content Perceptual Quality Assessment

  • Controlled user studies have examined size, distance, motion, spatial alignment, depth cues, shadows, and occlusion, but require repeated participant assessment.This reliance on manual evaluation limits scalability for large-scale or iterative AR assessment.Source: §2.1 · p. 3

2.2 Automatic AR Content Quality Evaluation

  • Automatic AR evaluators include full-reference metrics and deep-learning methods, but often lack robustness across scenes and platforms or assess only holistic or narrow features.These limitations motivate feature-specific assessment using VLM semantic and visual reasoning.Source: §2.2 · p. 3

2.3 VLMs for Perceptual Reasoning in AR

  • AR VLM applications have focused on content generation and placement; earlier scene-level assessments were coarse and could fail on contextually integrated virtual content.The paper addresses this gap with detailed, factor-specific quality assessment.Source: §2.3 · p. 3

3 RateAR Dataset

RateAR is a feature-level benchmark of AR images and videos spanning varied environments and perceptual quality. Five annotators rated three visual factors, with reported inter-annotator reliability ranging from good to excellent.

3 RateAR Dataset

  • RateAR contains 321 AR images and 112 AR videos covering placement plausibility, size appropriateness, and shadow realism.Images came from Android smartphones, Apple Vision Pro, and HoloLens 2; videos were recorded with Meta Quest 3.Source: §3 · pp. 3–4
  • Some floating content receives a high placement-plausibility score when the scene context makes the apparent suspension appropriate.Examples include dynamic interactions such as a hand bouncing a virtual basketball.Source: §3 · p. 4
  • Five annotators produced 6,495 feature labels on a five-point quality scale, with mean opinion scores used as the final ratings.Annotators discussed edge cases and used consensus meetings to resolve disagreements.Source: §3 · p. 4
  • ICC(2,5) was .95 for placement plausibility, .94 for size appropriateness, and .90 for shadow realism.The authors characterize the reliability as excellent for placement and size and good to excellent for shadow realism.Source: §3 · p. 4

4 Methodology

The methodology uses factor-specific VLM prompts for perceptual assessment and a closed-loop pipeline for AR content adjustment. Prompts incorporate scene context, while the pipeline converts VLM recommendations into rendering changes.

4 Methodology

  • The methodology evaluates factor-specific AR quality against human judgment and proposes tailored prompts plus a VLM-based closed-loop adjustment framework.Separate assessment of each feature is intended to avoid cross-factor interference.Source: §4 · p. 4, §4.1 · p. 4

4.1 Prompt Design

  • Prompts score placement, size, and shadows independently on contextual 1–5 judgments, allowing justified floating, atypical sizes, and nonstandard shadows.Placement considers object type and interaction; size prompts use plausible subtypes and scene cues, while shadow assessment weighs the use case.Source: §4.1 · pp. 4–5

4.2 Autonomous Content Adjustment System

  • The adjustment pipeline captures AR media, obtains VLM quality predictions, and retrieves and applies recommended rendering parameters.Video input may be subsampled into images; post-processing maps extracted numerical recommendations to supported system parameters without human intervention.Source: §4.2 · p. 5

5 Experiments

Experiments compared commercial VLMs with fine-tuned quality-assessment baselines on RateAR, then tested VLM-guided content adjustment with participants. VLM assessments aligned with human ratings, and participant ratings and feedback supported improvements to placement and sizing.

5.1 Experimental Setting

  • Eleven VLMs were evaluated against fine-tuned image and video baselines on RateAR test sets; VLMs used zero-shot inference, while baselines used separate models for each quality factor.Baseline training used 100 images and 50 videos, with the remaining 221 images and 62 videos reserved for testing.Source: §5.1 · p. 5
  • Annotator scores were standardized, clipped to [−3, +3] when needed, scaled to [0, 100], and averaged into three MOS values per sample.The three MOS values correspond to the RateAR quality factors.Source: §5.1 · p. 6

5.2.1 Image-based Assessment.

  • 0.03–0.14 SRCC, 0.02–0.15 PLCC, and 0.05–0.19 KRCC: GPT flagship models except GPT-5.4-Mini numerically exceeded the best image baselines across factors.Table 1 reports image correlations between human scores and model predictions.Source: §5.2.1 · p. 6, Table 1 · p. 7

5.2 Performance Analysis

  • For images, placement reached 0.8336 SRCC while strongest shadow realism was 0.6075; video shadow correlations led, with the video set mostly containing correct-shadow and no-shadow states.For images, placement relies on relatively unambiguous geometric cues, while shadow ratings depend on subtler, context-sensitive photometric attributes.Source: §5.2.1 · p. 6, §5.2.2 · pp. 6–7

5.2.2 Video-based Assessment.

  • Up to 0.26 SRCC, 0.32 PLCC, and 0.26 KRCC: the strongest video VLMs numerically outperformed leading image VLMs across factors, likely benefiting from temporal and multi-view cues.The authors conclude that videos provide richer evidence for AR quality assessment.Source: §5.2.2 · p. 7, Table 1 · p. 7, Table 2 · p. 7

5.3 User Study Design

  • The IRB-approved study recruited 21 participants and tested placement and sizing in parking-lot, toy-city, and office-table scenarios using image and 10-second video trials.Participants rated each object before and, when applicable, after the VLM recommended an adjustment.Source: §5.3 · p. 7, §5.3.2 · p. 8, §5.3.3 · p. 8

5.4 User Study Results

  • 45/84 trials (53.6%) improved, 38 (45.2%) were unchanged, and one (1.2%) worsened after adjustment; placement and size pipelines also showed significant rating gains.A ceiling effect accounted for some unchanged ratings: 20/84 trials began at the maximum score. The VLM also struggled with partially occluded content.Source: §5.4.1 · pp. 8–9

5.4.1 Adjustment Performance.

  • 19/21 participants (90.5%) agreed that the system improved placement or size quality; 61.9% were satisfied with image-based timeliness versus 33.3% with video-based timeliness.Processing subsampled video frames measurably increased recommendation latency.Source: §5.4.2 · p. 9

6 Ablation Study

The ablation compares primitive, contextual, and few-shot prompts for three AR quality features, using RateAR images to measure alignment with human ratings. Contextual prompts generally align best, while additional few-shot examples offer limited extra benefit.

6.1 Prompt Engineering

  • The study compares three prompt strategies that vary the complexity and information supplied to the VLM.The strategies are primitive, contextual, and few-shot prompting.Source: §6.1 · p. 9
  • Primitive prompts give concise task instructions; contextual prompts add AR examples and targeted visual cues, while few-shot prompts add three scored examples to contextual instructions.The few-shot examples represent poor quality (score 1), exemplar quality (score 5), and an intermediate case (score 3).Source: §6.1 · p. 9

6.2 Ablation Setup

  • The ablation uses GPT-o3, GPT-5, and GPT-5.4 on 221 RateAR test images, with separate prompts and few-shot examples for placement, size, and shadow realism.Primitive and contextual trials are zero-shot; few-shot examples come from outside the test set, and SRCC, PLCC, and KRCC measure alignment.Source: §6.2 · p. 9, §6 · p. 9

6.3 Prompting Strategy Analysis

  • 0.01 ∼ 0.15 SRCC, 0.01 ∼ 0.13 PLCC, and 0.01 ∼ 0.11 KRCC gains were observed for GPT-o3 with contextual or few-shot prompts versus primitive prompts.These ranges cover the three evaluated factors; GPT-5.4’s shadow-realism PLCC and KRCC were marginally higher with primitive prompts, by 0.01 each.Source: §6.3 · p. 9
  • Contextual prompts lead in 5 of 9 feature-metric pairings, compared with 2 for few-shot prompting; the results suggest added examples can plateau or degrade performance.The table reports correlations between human scores and GPT-family flagship predictions on RateAR images.Source: §6.3 · pp. 9–10, Table 3 · p. 10

7 Limitations and Future Work

The paper identifies limitations in RateAR’s media and feature coverage, the deployed prototype’s capabilities and latency, and the user-study design. It proposes broader datasets, more capable adjustment strategies, and alternative study designs as future work.

Dataset Composition

  • RateAR’s image and video samples differ in scene composition, source, and factor realization, limiting direct comparisons; it also covers only a subset of immersion-related visual features.Future work proposes better-matched image-video counterparts and additional features such as occlusion and object clipping.Source: §7 · p. 10

Prototype Scope

  • The proof-of-concept system performs one adjustment step for placement and size, omitting shadow refinement because assessment reliability is lower and a robust adjustment pipeline is lacking.Planned work includes investigating salient shadow attributes and multi-step adjustment strategies.Source: §7 · p. 10
  • Near-real-time adjustment remains out of scope because VLM inference is the primary bottleneck, with mean latencies of 4.43 s and 5.84 s for image placement and size, and 7.63 s and 9.62 s for video.The paper proposes investigating fine-tuned edge VLMs to avoid latency-intensive cloud inference.Source: §7 · p. 10

User Study Design

  • The 21-participant study uses a fixed before-and-after order that may introduce expectation bias, and some initially high-quality scenes create ceiling effects that reduce observable room for improvement.The authors propose larger, more diverse participant pools, controlled initial scene quality, and alternative study designs.Source: §7 · p. 10

8 Conclusion

The paper introduces RateAR and evaluates commercial VLMs for three AR visual features, finding that their predictions closely track human judgments. It also builds an adjustment pipeline and evaluates it in a 21-participant user study.

8 Conclusion

  • RateAR comprises 321 AR images and 112 AR videos from physical environments, and supports evaluation of eleven commercial VLMs on placement, size, and shadow realism.The reported VLM-human Spearman correlation reaches 0.8695.Source: §8 · p. 10
  • VLM predictions closely track human subjective judgments, with Spearman’s rank-order correlation reaching 0.8695.Source: §8 · p. 10
  • The authors develop a VLM-driven autonomous adjustment pipeline and validate its utility in an IRB-approved user study with 21 participants.The paper concludes that VLMs can provide perception-aware feedback that reduces reliance on labor-intensive user studies and brittle geometry reconstruction.Source: §8 · p. 10

A.1 VLM Content Adjustment Prompts

The appendix prompts ask VLMs to assess AR object placement and size, then return numeric adjustment parameters when needed.

A.1.1 Placement Adjustment Task Prompt.

  • The placement prompt rates whether an object is grounded or aligned, while allowing floating when appropriate for its type.It considers attached shadows, conceptual visualization, and interactions with forces or other objects.Source: §A.1.1 · p. 13
  • Placement is rated from 1 for clearly inappropriate floating or misalignment to 5 for proper grounding, alignment, or reasonable floating.Source: §A.1.1 · p. 13
  • For placement scores below 5, the prompt requests applicable movement directions and two offsets in meters, each ranging from -1 to 1.The first number specifies left-right movement and the second up-down; zero for both indicates proper placement.Source: §A.1.1 · p. 13

A.1.2 Size Adjustment Task Prompt.

  • The size prompt judges whether an object has a plausible category subtype and whether its scale fits surrounding references, affordances, and context.Atypical size may still be appropriate in context, such as in a surreal museum.Source: §A.1.2 · p. 13
  • Size appropriateness uses a 1–5 scale, from clearly too large or too small to normal or reasonably non-standard size.Source: §A.1.2 · p. 13
  • For size scores below 5, the prompt requests an increase or decrease and one scale adjustment from 0.01 to 2; 1 means retain the current scale.Values above 1 enlarge the content, while values below 1 shrink it.Source: §A.1.2 · p. 13

A.2 Extended RateAR Baseline and VLM Performance Analysis

The appendix documents confidence intervals and significance testing for the main correlation results, including multiple-test adjustments.

A.2 Extended RateAR Baseline and VLM Performance Analysis

  • Correlation coefficient 95% confidence intervals and significance p-values were computed using Pingouin.Tables A1 and A2 provide the corresponding image results; Tables A3 and A4 provide the video results.Source: §A.2 · p. 13
  • Image-result p-values were adjusted column-wise with Bonferroni correction for 18 tests.The adjustment is p_adj = min(1, p_raw * m_images), with m_images = 18.Source: §A.2 · p. 13
  • RateAR video coefficient p-values were adjusted using m_video = 17.The passage does not specify a separate formula for the video adjustment.Source: §A.2 · p. 13

A.3 Extended User Study Analysis

The appendix reports paired pre–post user-study tests and supplies supplementary analyses of primitive and few-shot prompts. It also documents confidence intervals and adjusted significance tests for the extended prompting analysis.

A.3 Extended User Study Analysis

  • All paired Wilcoxon tests had p-values below .05; all except the image-based size test were below .01.The tests cover pre–post placement and size scores from image- and video-based pipelines.Source: §A.3 · p. 13, Table A5 · p. 15

A.4 Ablation Study: Primitive and Few-Shot Prompts

  • The ablation appendix supplies primitive and few-shot prompts, while noting that contextual prompts are already given in Section 4.1.The primitive and few-shot prompt text is provided separately.Source: §A.4 · p. 13

A.4.1 Primitive Prompts.

  • Primitive prompts ask for separate 1–5 judgments of placement, size, and shadow appropriateness.The three prompts request a final response in the format “Score: [your rating].”Source: §A.4.1 · p. 13
  • Tables A1–A4 provide confidence intervals and significance p-values for the image and video correlations reported in Tables 1 and 2.The image tables correspond to Table 1; the video tables correspond to Table 2.Source: Table A1 · p. 14, Table A2 · p. 14, Table A3 · p. 15

A.3 Extended User Study Analysis

  • Figure A1 encodes pre–post placement and size changes for image and video pipelines as improvement, no change, or degradation.The caption describes the score categories but does not state the direction or magnitude of observed outcomes.Source: Figure A1 · p. 16

A.4.2 Few-Shot Prompts.

  • Few-shot prompts give three scored AR-scene examples for placement, size, and shadow judgments.The prompts ask the VLM to use the examples when rating a target scene.Source: §A.4.2 · pp. 17–18

A.5 Extended Prompting Strategy Analysis

  • For image correlations in Table 3, Tables A6 and A7 report confidence intervals and significance p-values, with Bonferroni adjustment over nine tests.The adjustment is column-wise.Source: §A.5 · p. 18, Table A6 · p. 19, Table A7 · p. 19
Loading 2610.00677v1…