PaperlayerView exact revision on arXiv

Source-linked AI summary

When Visual Quality Misleads: Intent Recognition under Rendered Avatar Distortions

Ning-Hsuan Chang, Kai-Siang Ma, Yu-Chih Chen

arXiv:2609.27560v1cs.MMcs.CVcs.HC

TL;DR

Avatar-streaming systems commonly use visual quality as a proxy for communicative success, although those targets may diverge. This study tests metrics against behavioral judgments across controlled avatar distortions, finding frequent cases of misleading quality and limited metric alignment with intent. The results motivate treating appearance and communication as complementary objectives within the tested protocol.

  • Problem

    Whether conventional IQA/VQA metrics predict whether observers recover rendered avatars’ intended actions remains open, despite visual fidelity and communicative success being distinct evaluation targets.

    Source: §1 · p. 2

  • Method

    A controlled study of 59 participants’ judgments across pristine and distorted avatar renderings derives IQS from recognition correctness and confidence to evaluate objective metrics.

    Source: §1 · p. 2, §2 · p. 2

  • Results

    31 of 126 distorted cells (24.6%) showed Misleading Quality, and metric alignment with IQS remained limited, with Re-IQA reaching PLCC = 0.4435 at λ=0.5.

    Source: §3 · p. 3

  • Takeaways & Limitations

    Visual fidelity and communicative effectiveness should be treated as complementary rather than interchangeable objectives in rendered-avatar streaming.

    Source: §4 · p. 3

    Conclusions are restricted to the evaluated protocol, including desktop free-orbit viewing, canonical-view metrics, controlled distortions, and nine retained avatar–action units.

    Source: §4 · p. 3

Abstract

from arXiv · show

Avatar-streaming systems are commonly evaluated with image and video quality assessment (IQA/VQA) metrics, implicitly treating visual fidelity as a proxy for communicative success. We test this assumption through a controlled behavioral study of rendered 3D avatars across a pristine condition and fourteen geometric, photometric, temporal, and combined distortions. Fifty-nine participants contributed 2,688 judgments of perceived action, response confidence, and visual quality. We identify Misleading Quality in this dataset as distorted renderings that retain above-average perceived quality but yield below-average action-recognition accuracy. We also derive an Intent Quality Score (IQS) combining recognition correctness and confidence as the behavioral target for objective metrics. Among 126 distorted content--condition cells, 31 (24.6%) exhibited Misleading Quality; temporal and geometric distortions showed the highest rates, at 50.0% and 31.1%, respectively. The results reveal a quality--accuracy dissociation where distortion families affect appearance and communication differently. Across 24 direct-scoring IQA/VQA metrics and three supervised feature-regression baselines, alignment with IQS remained limited; at $\lambda=0.5$, the best leave-one-content-out baseline reached PLCC $=0.4435$. Under this controlled protocol, visual fidelity alone is insufficient for avatar communication, motivating intent-aware quality assessment and streaming objectives.

1 Introduction

Avatar-streaming evaluation often uses visual fidelity as a proxy for communicative success, although observers must understand intended actions and visual and communicative outcomes can diverge. This study tests whether conventional IQA/VQA metrics predict action recognition and benchmarks their intent sensitivity.

1 Introduction

  • Visual fidelity and communicative success are distinct targets because distortions can leave renderings acceptable while corrupting pose, motion, or geometric cues carrying meaning.Avatar-streaming systems also balance visual fidelity against bandwidth and rendering cost.Source: §1 · p. 2
  • The study tests whether conventional IQA/VQA metrics predict whether observers recover rendered avatars’ intended actions, a gap left open by prior quality and graphics-perception research.It contributes a controlled behavioral protocol, an operational characterization of Misleading Quality, and a benchmark of existing metrics’ intent sensitivity.Source: §1 · p. 2
  • The behavioral study gathers 2,688 judgments from 59 participants across ten avatar–action units and pristine plus fourteen distorted conditions.Participants report perceived action, confidence, and visual quality; recognition correctness and confidence are combined into an Intent Quality Score (IQS).Source: §1 · p. 2

2 Controlled Study and Evaluation Protocol

The protocol tests action understanding and perceived quality across controlled avatar distortions, then evaluates objective metrics against a confidence-weighted behavioral target. Metric correlations are measured under a fixed-view, leave-one-content-out setup.

2 Controlled Study and Evaluation Protocol

  • Five stylized avatars paired with two skeletal animations yielded ten content units, each rendered in one pristine and fourteen distorted conditions for 150 stimuli.Distortions span geometric, photometric, temporal, and combined families and are controlled rendering-level approximations, not outputs of specific production codecs.Source: §2 · p. 2
  • Fifty-nine participants contributed 2,688 judgments; analyses retained nine contents after excluding E5 for failing the pristine-response consensus threshold.Participants selected perceived actions, rated confidence, and rated visual quality; the retained set comprised 135 cells, including 126 distorted cells.Source: §2 · p. 2
  • IQS combines recognition correctness and normalized confidence, penalizing confident misidentification, and is averaged within each content–condition cell.Main results use λ=0.5; supplementary analyses report the same conclusions across λ values from 0 to 1.Source: §2 · p. 2
  • The evaluation compares 24 direct-scoring FR/NR-IQA and FR/NR-VQA metrics with three supervised feature-regression baselines.Full-reference methods use the corresponding pristine rendering; still-image analyses exclude the two low-frame-rate conditions.Source: §2 · pp. 2–3
  • Feature regressors use fixed-seed leave-one-content-out validation, while objective metrics use canonical-view images or videos rather than participants’ freely orbited views.Thus, reported correlations characterize this controlled protocol, not unrestricted interactive or HMD perception.Source: §2 · p. 3

3 Results

The results show a quality–accuracy dissociation: some distorted renderings remain visually acceptable despite reduced action recognition, while other visible degradations preserve recognition. Objective metrics align only weakly with IQS.

3 Results

  • 31 of 126 distorted cells (24.6%) met the study’s Misleading Quality thresholds of above-mean visual quality and below-mean recognition accuracy.Temporal low-frame-rate distortions had the highest prevalence (9/18, 50.0%), followed by geometric distortions (14/45, 31.1%).Source: §3 · p. 3
  • The opposite quality–accuracy quadrant also contained visibly salient degradations that reduced perceived quality while preserving recognition, so appearance and communicative failure were not monotonically coupled.Source: §3 · p. 3
  • Across λ values, no direct-scoring IQA/VQA method exceeded |PLCC| = 0.3237; at λ=0.5, Re-IQA was strongest among feature-regression baselines, with PLCC = 0.4435 and SRCC = 0.4501.Confidence intervals are broad, so fine-grained differences among baselines are descriptive rather than reliable rankings.Source: §3 · p. 3
  • Split-half IQS agreement reached PLCC = 0.6381, above all objective methods, indicating that target unreliability alone cannot explain limited metric performance.IQS is reproducible, though not noise-free.Source: §3 · p. 3

4 Discussion, Limitations, and Conclusion

The findings support treating visual fidelity and communicative effectiveness as complementary objectives for rendered-avatar streaming. The conclusions remain bounded by the evaluated viewing protocol, stimuli, and confidence-weighted target.

4 Discussion, Limitations, and Conclusion

  • Visual fidelity and communicative effectiveness should be complementary, not interchangeable, objectives for rendered-avatar streaming.Temporal and geometric degradations more often reduced recognition despite relatively high perceived quality, and better IQA/VQA scores need not protect intent-relevant cues.Source: §4 · p. 3
  • A practical direction is intent-aware evaluation or content-adaptive protection of communication-relevant cues, but this study does not provide a deployable task-aware quality metric.Source: §4 · p. 3
  • Conclusions are limited to monoscopic desktop free-orbit viewing, canonical-view metrics, controlled non-codec distortions, nine retained avatar–action units, and constrained action choices.Action and character effects cannot be disentangled, distortion severities were not perceptually matched, and IQS is a pragmatic rather than complete model of understanding.Source: §4 · p. 3

Appendix

Table 2 is identified as the inventory of avatar–action content units.

Appendix

  • Table 2 summarizes the avatar–action content units used in the study.Source: Table 2 · p. 5

A Stimuli, Distortions, and Evaluation Details

The supplementary material describes controlled rendering-level distortions implemented with Three.js/WebGL, not outputs of production codecs.

A Stimuli, Distortions, and Evaluation Details

A.1 Avatar Assets and Actions

The study paired five stylized avatars with motion-capture animations to form avatar–action content units spanning communicative behaviors.

A.1 Avatar Assets and Actions

  • Five stylized 3D avatars were each paired with two Mixamo motion-capture animations, yielding ten avatar–action content units.Source: §A.1 · p. 5
  • The action set covers greeting, affective, and socially communicative behaviors.Source: §A.1 · p. 5
  • Ground-truth action labels came from the modal response under pristine ORIG, and actions required at least 0.6 pristine-condition consensus.E5 was excluded and E6 retained under this criterion.Source: §A.1 · p. 5

A.2 Distortion Conditions

This appendix section details the controlled distortion presets and the rendering and evaluation procedures used to assess metric correspondence with IQS.

A.2 Distortion Conditions

  • Combined presets were treated as named conditions; their operations were not varied factorially, so comparisons do not estimate component-level effects.Source: §A.2 · p. 5
  • Table 3 summarizes direct-scoring metric correlations with IQS by family, with rows ordered by PLCC; Figure 3 shows absolute PLCC and SRCC sensitivity to λ.Figure 3 covers 24 direct-scoring metrics and three feature-regression baselines, emphasizing the strongest methods near λ=0.5.Source: Table 3 · p. 5, Figure 3 · p. 6
  • Participants chose among three to five action-specific options, so raw recognition accuracy was not perfectly chance-equated across actions.The same observed cell-level accuracy definition was applied across conditions.Source: §A.2 · p. 5
  • Still images were rendered at 1024×1024 at 1.0 s; videos used ten 512×512 frames sampled over 0.50–1.50 s and encoded as 1-s, 10-fps H.264 clips.Source: §A.2 · p. 5
  • Fixed-seed leave-one-content-out validation held out all cells for one avatar–action unit, preventing train–test overlap across distorted versions of the same content.Source: §A.2 · p. 6
  • The study comprised 117 NR-IQA cells, 108 FR-IQA distorted–reference pairs, 135 NR-VQA cells, and 126 FR-VQA distorted–reference pairs; fixed-seed leave-one-content-out validation held out all cells associated with one avatar–action content unit.Source: §A.2 · p. 6

B IQS Robustness and Statistical Reliability

IQS is a pragmatic confidence-weighted task score, not a formal signal-detection measure; increasing λ raises the penalty for confident misidentification. Metric associations with IQS remain limited across the tested λ values, with uncertainty assessed using content-block resampling and behavioral split-half agreement as a reliability reference.

B IQS Robustness and Statistical Reliability

  • Increasing λ assigns a progressively larger penalty to confident misidentification; at λ = 0, incorrect responses score zero.IQS is used as a pragmatic confidence-weighted task score rather than a formal signal-detection-theory measure.Source: B IQS Robustness and Statistical… · p. 6
  • Across the full sensitivity sweep, feature regression reached |PLCC| = 0.4604, while no direct-scoring metric exceeded |PLCC| = 0.3237.The absolute correlations compare association magnitude despite differences in native score direction.Source: B IQS Robustness and Statistical… · p. 6
  • The limited association between existing metrics and IQS persisted across the tested λ values, rather than depending on the primary choice of λ = 0.5.The reported sensitivity analysis uses absolute correlations because metrics differ in native score direction.Source: B IQS Robustness and Statistical… · p. 6
  • Correlation uncertainty was estimated with 10,000 content-block bootstrap resamples, keeping cells from sampled content units grouped.This preserves within-content dependence across distortion conditions instead of treating individual cells as independent.Source: B IQS Robustness and Statistical… · p. 6
  • Split-half agreement was estimated by splitting responses within each content–condition cell and correlating independently recomputed cell-level IQS vectors.The study treats this agreement as a behavioral reliability reference, not a literal upper bound for every predictor.Source: B IQS Robustness and Statistical… · p. 6

Related papers

Loading 2609.27560v1…