Source-linked AI summary
Bridge the Gap Between VQA and Human Behavior on Omnidirectional Video: A Large-Scale Dataset and a Deep Learning Model
Chen Li, Mai Xu, Xinzhe Du, Zulin Wang
TL;DR
Omnidirectional VQA methods do not account for head and eye movements, which shape the regions observers view and perceive clearly. The paper builds a dataset pairing subjective scores with HM/EM data and incorporates those behaviors into a deep learning VQA model. It reports that the model significantly improves state-of-the-art performance, while dataset analysis finds subjective quality related to HM and EM.
Problem
Existing omnidirectional VQA approaches do not consider HM and EM, despite their roles in determining the viewed and clearly perceived regions, and no dataset combines subjective scores with corresponding HM/EM data.
Source: §1 · p. 2Method
The paper establishes VQA-ODV with subjective scores and HM/EM data for 600 sequences and develops a deep learning VQA model integrating HM and EM maps.
Source: §1 · p. 2, §5.1 · p. 7, §6 · p. 8Results
Takeaways & Limitations
Abstract
from arXiv · showhide
Omnidirectional video enables spherical stimuli with the $360 \times 180^ \circ$ viewing range. Meanwhile, only the viewport region of omnidirectional video can be seen by the observer through head movement (HM), and an even smaller region within the viewport can be clearly perceived through eye movement (EM). Thus, the subjective quality of omnidirectional video may be correlated with HM and EM of human behavior. To fill in the gap between subjective quality and human behavior, this paper proposes a large-scale visual quality assessment (VQA) dataset of omnidirectional video, called VQA-OV, which collects 60 reference sequences and 540 impaired sequences. Our VQA-OV dataset provides not only the subjective quality scores of sequences but also the HM and EM data of subjects. By mining our dataset, we find that the subjective quality of omnidirectional video is indeed related to HM and EM. Hence, we develop a deep learning model, which embeds HM and EM, for objective VQA on omnidirectional video. Experimental results show that our model significantly improves the state-of-the-art performance of VQA on omnidirectional video.
1 INTRODUCTION
Omnidirectional VQA methods do not account for how head and eye movements shape the portions of a video observers see clearly. The paper links subjective quality with those behaviors and introduces a dataset and model incorporating them.
1 INTRODUCTION
- Existing VQA approaches do not account for human viewing behavior, although HM determines the viewport and EM determines what is clearly perceived within it.The paper notes that sequences with similar PSNR can have different DMOS because subjects rate the regions they see.Source: §1 · pp. 1–2, Figure 1 · p. 1
- No prior dataset combined subjective VQA scores with corresponding HM and EM data to investigate their relationship for omnidirectional video.The paper identifies this as a gap despite existing datasets collecting HM or EM data.Source: §1 · p. 2
- The paper establishes VQA-ODV with subjective scores and HM/EM data for 600 sequences, then finds subjective scores related to those behaviors.The sequences vary in content, duration, resolution, compression, and map projection.Source: §1 · p. 2
- A deep learning VQA model integrates HM and EM, and the authors report that it significantly advances state-of-the-art performance.The authors present this result as evidence for considering human behavior in omnidirectional video quality assessment.Source: §1 · p. 2
2 RELATED WORKS
Prior work studies subjective quality, projection-aware objective metrics, and human viewing behavior, but the paper identifies a missing link: joint analysis of quality and HM/EM data.
2 RELATED WORKS
- Subjective VQA work includes testbeds, experiments on scene and device factors, assessment procedures, and O-DMOS/V-DMOS score processing.O-DMOS and V-DMOS account for consistency of viewing direction across subjects.Source: §2 · p. 2
- Objective omnidirectional metrics modify PSNR through projection-aware weighting or resampling, but the cited approaches do not consider human viewing behavior.The approaches include WS-PSNR, AW-SPSNR, S-PSNR, and CPP-PSNR.Source: §2 · p. 2
- Prior datasets and models collect or predict HM and EM, yet the paper reports no prior work linking viewing behavior to visual quality or dataset combining both with subjective VQA scores.The authors establish VQA-ODV and analyze its data to address that gap.Source: §2 · p. 2
3 DATASET ESTABLISHMENT
VQA-ODV contains reference and impaired omnidirectional sequences, subjective quality ratings, and synchronized HM/EM measurements. The dataset is organized for collection across diverse content and includes defined score and behavior-data formats.
3.1 Omnidirectional sequences
- VQA-ODV contains 60 diverse reference sequences spanning 4K–8K resolution, varied content, and 10–23-second durations.The references use equirectangular projection and are divided into 10 groups with varied content and resolution.Source: §3.1 · pp. 2–3, Figure 2 · p. 3
- Each reference yields nine impaired sequences from three compression levels and three projections, producing 540 impaired sequences.The compression levels use H.265 at QP 27, 37, and 42; projections include ERP, RCMP, and TSP.Source: §3.1 · p. 3
3.2 Subjective data collection
- The collection protocol uses calibrated training and randomized single-stimulus testing, with subjects rating each sequence while HM/EM are captured alongside playback.The ratings use a continuous 0–100 quality scale, and behavior measurements are aligned with video frames.Source: §3.2 · pp. 3–4
- The experiment involved 221 subjects divided across 10 sequence groups, with at least 20 valid subjects remaining in each group after rejection.Each subject watched one group to avoid eye fatigue.Source: §3.2 · p. 4
3.3 Subjective data formats
- The dataset provides MOS and DMOS scores; DMOS represents subjective quality degradation for impaired video, while reference-sequence DMOS is zero.MOS for a sequence is computed from the raw ratings of its valid viewers.Source: §3.3 · p. 4
- Each HM/EM sample records timestamp, head pitch/yaw/roll, eye x/y coordinates, and an eye-data validity flag.Eye coordinates are normalized to [0, 1]; HM data are valid, while eye data may be invalid, for example during a blink.Source: §3.3 · p. 4
4 DATA ANALYSIS
Across the analysis, subjective quality varies more with bitrate than projection, while human viewing patterns are consistent and cover only part of the sphere. Weighting PSNR with HM and EM data improves its correlation with subjective scores.
4.1 Analysis on VQA Results
- Higher bitrates yield better subjective quality than lower bitrates, while TSP is slightly better at matched bitrates and projection matters little at high bitrates.The analysis compares DMOS across projection types and bitrate levels.Source: §4.1 · p. 5, Table 1 · p. 5, Figure 6 · p. 5
- S-PSNR performs better than traditional PSNR and SSIM on omnidirectional video, where traditional metrics are less effective than on 2D video.The comparison uses correlations with subjective DMOS; the tables report metric performance on omnidirectional and 2D video.Source: §4.1 · p. 5, Table 2 · p. 6, Table 3 · p. 6
4.2 Analysis on Human Behavior
- HM weight maps show high agreement across subject subgroups, with average PCC 0.8688; EM maps show similar consistency.The comparison covers the 60 reference sequences.Source: §4.2 · p. 5, Figure 7 · p. 5
- All subjects view less than 65% of each omnidirectional frame, yet this partial viewport region determines perceived quality.The viewport percentage is averaged over frames for each reference sequence.Source: Figure 8 · p. 5, §4.2 · p. 6
4.3 Impact of Human Behavior on VQA
- The analysis forms perceptual weights from subject HM viewports and EM positions, including Gaussian filtering of eye positions within the viewport.It defines individual and overall HM weights as well as individual EM weights for weighted PSNR assessment.Source: §4.3 · p. 6
- Incorporating HM data significantly improves PSNR performance, and EM maps further improve it over HM-weighted PSNR.The comparison is reported for correlations between PSNR metrics and DMOS over VQA-ODV.Source: Table 2 · p. 6, §4.3 · p. 7
5 DEEP LEARNING BASED VQA MODEL
The model integrates head- and eye-movement information into deep-learning VQA for omnidirectional video. On the test set, it outperforms the compared VQA approaches using predicted HM and EM maps.
5.1 Method
- The model samples impaired patches according to HM weights, then combines their local VQA scores using normalized EM weights to estimate sequence quality.Each sampled patch is 112 × 112 pixels, and its corresponding error map is also input to the CNN component.Source: §5.1 · p. 7
- Training uses a three-term loss comprising score error, total-variation smoothing of sensitivity maps, and L2 regularization of network parameters.The score-error term compares predicted VQA scores with ground-truth DMOS scores.Source: §5.1 · p. 7
5.2 Evaluation
- The evaluation tests 108 impaired sequences from 12 reference sequences and uses predicted HM and EM maps because those data are unavailable for objective assessment in practice.The remaining 432 impaired sequences from 48 reference sequences form the training set.Source: §5.2 · pp. 7–8
- PCC and SRCC rise from 0.69 and 0.73 for DeepQA to 0.77 and 0.80 with predicted HM and EM, while RMSE and MAE fall by 1.20 and 0.95.The comparison is over 108 test sequences and includes four other VQA approaches.Source: Table 4 · p. 8, §5.2 · p. 8
- With ground-truth rather than predicted HM and EM maps, PCC and SRCC improve from 0.77 and 0.80 to 0.81 and 0.83, respectively.The authors identify map-prediction accuracy as influencing model performance; these values represent the model's upper-bound performance.Source: §5.2 · p. 8, Table 4 · p. 8
6 CONCLUSION AND FUTURE WORK
The paper presents a 600-sequence omnidirectional-video VQA dataset with subjective scores and HM/EM data, and a deep-learning model that integrates those maps. The model's objective scores correlate more closely with subjective scores than the compared state-of-the-art approaches.
6 CONCLUSION AND FUTURE WORK
- The proposed deep-learning model integrates HM and EM maps and estimates objective scores that are more correlated with subjective scores than other state-of-the-art approaches.The VQA-ODV dataset contains subjective scores and HM/EM behavior data for 600 omnidirectional sequences; applying the approach to video-processing tasks is proposed as future work.Source: §6 · p. 8