Source-linked AI summary
Exploring Video Quality Assessment on User Generated Contents from Aesthetic and Technical Perspectives
Haoning Wu, Erli Zhang, Liang Liao, Chaofeng Chen, Jingwen Hou, Annan Wang, Wenxiu Sun, Qiong Yan, Weisi Lin
TL;DR
UGC-video quality opinions are ambiguous because they can reflect technical distortions and aesthetic preferences. The paper collects multi-perspective opinions in DIVIDE-3k and develops DOVER and DOVER++ to model overall and single-perspective quality, achieving state-of-the-art UGC-VQA performance.
Problem
UGC-VQA lacks a clear objective because human opinions on diverse videos may reflect both technical distortions and aesthetic preferences.
Method
The paper collects 450,000 multi-perspective opinions on 3,590 videos and proposes DOVER’s two perspective-specific branches, followed by DOVER++ for single-perspective evaluation.
Results
DOVER achieves state-of-the-art performance across UGC-VQA datasets, while DOVER++ provides quality evaluations from a single aesthetic or technical perspective.
Takeaways & Limitations
UGC-video quality assessment should account for both aesthetic and technical perspectives, while separate perspective predictions support technical, recommendation, and personalized applications.
Takeaways & Limitations
Some perceptual factors, including brightness and motion blur, relate to both aesthetic and technical perspectives and therefore remain in both branches.
Abstract
from arXiv · showhide
The rapid increase in user-generated-content (UGC) videos calls for the development of effective video quality assessment (VQA) algorithms. However, the objective of the UGC-VQA problem is still ambiguous and can be viewed from two perspectives: the technical perspective, measuring the perception of distortions; and the aesthetic perspective, which relates to preference and recommendation on contents. To understand how these two perspectives affect overall subjective opinions in UGC-VQA, we conduct a large-scale subjective study to collect human quality opinions on overall quality of videos as well as perceptions from aesthetic and technical perspectives. The collected Disentangled Video Quality Database (DIVIDE-3k) confirms that human quality opinions on UGC videos are universally and inevitably affected by both aesthetic and technical perspectives. In light of this, we propose the Disentangled Objective Video Quality Evaluator (DOVER) to learn the quality of UGC videos based on the two perspectives. The DOVER proves state-of-the-art performance in UGC-VQA under very high efficiency. With perspective opinions in DIVIDE-3k, we further propose DOVER++, the first approach to provide reliable clear-cut quality evaluations from a single aesthetic or technical perspective. Code at https://github.com/VQAssessment/DOVER.
1. Introduction
UGC-VQA opinions are ambiguous because they may reflect technical distortions, aesthetic preferences, or both. The paper addresses this ambiguity with a multi-perspective subjective database and a two-branch evaluator.
- UGC-VQA seeks to predict human quality-of-experience opinions for diverse in-the-wild user-generated videos, but those opinions may reflect different perspectives.
- Technical VQA measures distortions such as blur and artifacts, whereas aesthetic judgments concern semantic content and composition.
- The study collects opinions from aesthetic, technical, and overall perspectives, plus reasoning about each perspective’s influence, across 3,590 videos and 450,000 opinions.
- DIVIDE-3k shows that human UGC-video quality perception is broadly and inevitably affected by both aesthetic and technical perspectives.
- DOVER uses View Decomposition and two perspective-specific branches with tailored inputs, regularization, and pre-training to assess UGC-video quality.
- DOVER achieves state-of-the-art performance across UGC-VQA datasets, while DOVER++ provides single-perspective quality predictions for technical, aesthetic, or personalized applications.
2. Related Works
UGC-VQA differs from traditional VQA because it evaluates unconstrained real-world videos with unique content and no reference videos, making quality judgments less clear-cut.
- UGC-VQA databases use real-world videos from sources such as direct photography, social media, and YouTube rather than traditional reference-based video collections.
- Unique content, varied production quality, and no-reference crowdsourced studies make UGC-video quality assessment more challenging and less clear-cut than traditional VQA.
3. The DIVIDE-3k Database
DIVIDE-3k is a multi-perspective UGC-VQA database built from 3,590 videos and 450,000 in-lab human opinions. Its study measures aesthetic, technical, overall, and perspective-impact judgments to clarify how quality is perceived.
- Database construction: DIVIDE-3k contains 3,590 UGC videos and 450,000 human opinions collected through an in-lab multi-perspective subjective study.
- Database construction: The videos are drawn mainly from YFCC-100M and Kinetics-400, with 3,270 sampled to match spatial, temporal, and semantic distributions and 320 LSVQ videos for calibration.
- In-lab subjective study: Thirty-five trained annotators participate in Training, Testing, and Annotation, with videos downloaded locally to avoid transmission errors.
- In-lab subjective study: The study records aesthetic, technical, overall, and subjective-reasoning scores, including technical-impact proportions selected from [0, 0.25, 0.5, 0.75, 1].
- Analysis and observations: DIVIDE-3k analyzes correlations between aesthetic, technical, and overall quality ratings and distributions of overall MOS and technical impact.
- Analysis and observations: A weighted combination of aesthetic and technical opinions approximates overall quality better than either perspective alone, and overall DIVIDE-3k scores correlate more strongly with existing LSVQ scores.
- Analysis and observations: Subjective reasoning finds mean technical impact varying from 0.364 to 0.698 across videos, explicitly confirming that both perspectives affect human quality assessment.
4. The Approaches: DOVER and DOVER++
DOVER separates aesthetic and technical information into dedicated views and branches, then uses perspective-aware supervision and fusion to predict overall and perspective quality.
- View Decomposition: View Decomposition separates videos into an Aesthetic View for aesthetic perception and a Technical View for technical quality.
- Aesthetic Branch: The aesthetic branch uses spatial downsampling and sparse frame sampling to preserve semantics and composition while reducing technical influences.
- Technical Branch: The technical branch uses randomly cropped, stitched fragments that retain technical distortions while disrupting content and composition.
- Learning Objectives: DOVER applies Limited View Biased Supervisions to train each branch against overall MOS, exploiting view-specific inductive biases.
- Learning Objectives: DOVER++ adds Direct Supervisions from DIVIDE-3k perspective opinions and fuses the objectives to learn overall and perspective predictions.
- Learning Objectives: The final overall prediction uses the subjective-study weighting Qpred = 0.428Qpred,A + 0.572Qpred,T.
5. Experimental Evaluation
The experiments evaluate whether DOVER branches learn their intended perspectives and whether fusion improves overall UGC-VQA quality, using perspective studies and benchmark setups.
- Evaluation Goals: The evaluation tests branch alignment with aesthetic and technical effects, fused overall prediction, ablations, and personalized quality evaluation.
- Implementation Details: Implementation uses separate aesthetic and technical views, distinct backbones, and different spatial and temporal sampling strategies for inference and training.
- Experimental Setup: DIVIDE-3k perspective evaluation compares weakly supervised DOVER and fully supervised DOVER++ against models without View Decomposition.
- Experimental Setup: The study includes qualitative visualizations of videos whose predicted aesthetic and technical scores diverge.
5.2. Evaluation on Two Perspectives
DOVER and DOVER++ produce branch predictions that align with corresponding human aesthetic and technical opinions, including on videos where the perspectives diverge.
- Quantitative Studies: DOVER shows stronger perspective preference than a no-decomposition baseline even without respective labels, supporting View Decomposition.
- Quantitative Studies: DOVER++ branches are each around 7% more correlated with corresponding opinions than with opinions from the other perspective.
- Qualitative Studies: Videos with better aesthetic scores show clear semantics but blur or artifacts, whereas videos with better technical scores are sharp but compositionally chaotic.
- Divergence Map: On LSVQ divergence maps, the aesthetic branch distinguishes chaotic from symmetric scenes while the technical branch detects blur, over-exposure, and compression errors.
- User Studies: Pairwise studies with 15 subjects found each perspective’s votes notably more agreed with the corresponding branch predictions.
5.3. Evaluation on Overall Quality Prediction
Across existing and newly collected UGC-VQA datasets, DOVER achieves strong overall-quality prediction, while DOVER++ and perspective modeling further improve performance.
- Existing UGC-VQA Datasets: DOVER outperforms state-of-the-art methods on LSVQ-trained intra-dataset evaluations, improving PLCC by up to 2.0%.
- Existing UGC-VQA Datasets: DOVER improves PLCC on FAST-VQA by 3.3% on KoNViD-1k, whose contents are more diverse.
- Existing UGC-VQA Datasets: On three smaller UGC-VQA datasets, DOVER reaches mean PLCC > 0.89 and improves over FAST-VQA by an average of 2.6% under the same training process.
- DIVIDE-3k: On DIVIDE-3k, DOVER achieves state-of-the-art performance, while DOVER++ further improves overall prediction using aesthetic and technical scores as auxiliary labels.
- DIVIDE-3k: In zero-shot and cross-dataset evaluation on DIVIDE-3k, DOVER outperforms other listed methods.
5.4. Ablation Studies
Ablations show that decomposing video views, separately supervising the two branches, and weighting their fusion improve DOVER, while LDS adds about 1% to DOVER++ performance.
- Effects of View Decomposition: View Decomposition performs much better than directly feeding original videos to both branches, demonstrating the strategy's effectiveness.
- Effects of View Decomposition: Separate supervision outperforms Feature Aggregation when the backbone and input remain unchanged.Feature Aggregation concatenates the two branches' features before regression.
- Effects of Subjectively-Inspired Fusion: Using only one branch decreases performance, while unweighted addition is less accurate than subjectively inspired fusion.The fusion ablation supports combining aesthetic and technical predictions with weights.
- Ablation Studies of DOVER++: Around 1% performance gain comes from combining LDS with LLVBS in DOVER++.LDS alone also outperforms LLVBS without direct MOS supervision.
- Ablation Studies of DOVER++: DOVER++'s objective ablation supports modeling UGC-VQA quality as a sum of aesthetic and technical perspectives.
5.5. Outlook: Personalized Quality Evaluation
Individuals weight aesthetic and technical quality differently when rating the same UGC video, motivating adaptive fusion for personalized quality evaluation.
- Personalized Quality Evaluation: Individuals differ in how strongly technical quality affects their overall ratings of the same aesthetically stronger but blurry, under-exposed video.Subjects who consider the technical perspective more tend to assign lower scores.
- Personalized Quality Evaluation: DOVER++ can adapt fusion weights to align quality predictions with opinions from different groups.
6. Conclusion
The study finds that aesthetic and technical perspectives both affect human quality opinions on UGC videos. DOVER and DOVER++ use this insight to improve overall assessment and decouple perspective-specific evaluations.
- Conclusion: Both aesthetic and technical perspectives impact human quality opinions in UGC-VQA.
- Conclusion: DOVER and DOVER++ aim to improve overall UGC-VQA performance while decoupling aesthetic and technical effects.The decoupled evaluators target scenarios requiring pure technical or aesthetic quality metrics.