PaperlayerView exact revision on arXiv

Source-linked AI summary

UGC-VQA: Benchmarking Blind Video Quality Assessment for User Generated Content

Zhengzhong Tu, Yilin Wang, Neil Birkbeck, Balu Adsumilli, Alan C. Bovik

arXiv:2005.14354v2cs.CVeess.IV

TL;DR

UGC videos require blind quality assessment because their authentic impairments are unpredictable, diverse, and often intermixed. The paper benchmarks leading BVQA models and builds VIDEVAL by selecting and fusing features from existing models, reporting state-of-the-art performance at substantially lower computational cost. It also establishes a reproducible UGC-VQA evaluation framework for future research.

  • Problem

    UGC videos contain unpredictable, complicated, and commingled degradations, creating a challenging need for accurate blind quality prediction.

  • Method

    The paper benchmarks leading BVQA models in a fixed evaluation framework and constructs VIDEVAL by selecting and fusing features from existing top-performing models.

  • Results

    VIDEVAL achieves state-of-the-art performance at considerably lower computational cost than other leading models.

  • Takeaways & Limitations

    The study provides a reliable benchmark for UGC-VQA and supports further research on efficient and deep learning-based video quality modeling.

Abstract

from arXiv · show

Recent years have witnessed an explosion of user-generated content (UGC) videos shared and streamed over the Internet, thanks to the evolution of affordable and reliable consumer capture devices, and the tremendous popularity of social media platforms. Accordingly, there is a great need for accurate video quality assessment (VQA) models for UGC/consumer videos to monitor, control, and optimize this vast content. Blind quality prediction of in-the-wild videos is quite challenging, since the quality degradations of UGC content are unpredictable, complicated, and often commingled. Here we contribute to advancing the UGC-VQA problem by conducting a comprehensive evaluation of leading no-reference/blind VQA (BVQA) features and models on a fixed evaluation architecture, yielding new empirical insights on both subjective video quality studies and VQA model design. By employing a feature selection strategy on top of leading VQA model features, we are able to extract 60 of the 763 statistical features used by the leading models to create a new fusion-based BVQA model, which we dub the \textbf{VID}eo quality \textbf{EVAL}uator (VIDEVAL), that effectively balances the trade-off between VQA performance and efficiency. Our experimental results show that VIDEVAL achieves state-of-the-art performance at considerably lower computational cost than other leading models. Our study protocol also defines a reliable benchmark for the UGC-VQA problem, which we believe will facilitate further research on deep learning-based VQA modeling, as well as perceptually-optimized efficient UGC video processing, transcoding, and streaming. To promote reproducible research and public evaluation, an implementation of VIDEVAL has been made available online: \url{https://github.com/tu184044109/VIDEVAL_release}.

I. INTRODUCTION

UGC videos create a difficult blind VQA problem because their authentic impairments are diverse, unpredictable, and often intermixed. The paper benchmarks leading BVQA models and proposes VIDEVAL, a feature-selected fusion model for efficient quality prediction.

  • Motivation: Legacy VQA databases use few pristine videos with controlled synthetic distortions, limiting their representation of real-world variation.These datasets typically apply only one or two synthetic distortions to each unique content.
  • Motivation: UGC-VQA datasets use consumer-generated videos with authentic impairments, unavailable references, and intermixed capture, processing, compression, and transmission artifacts.Compression is not necessarily the dominant quality factor in UGC videos.
  • Motivation: Unpredictable perceptual degradations make blind quality prediction for UGC consumer videos challenging.
  • Contributions: The paper benchmarks leading video quality models on several large-scale UGC-VQA databases and proposes the fusion-based VIDEVAL algorithm.VIDEVAL is created through feature selection from existing top-performing VQA models.
  • Contributions: The study reports that aggregating known models can achieve state-of-the-art performance for the UGC-VQA problem.

II. UGC-VQA DATABASES

The paper reviews large-scale UGC-VQA databases that differ in content sources, diversity, organization, and subjective-rating characteristics. It analyzes these differences using low-level visual features, coverage statistics, deep features, and MOS distributions.

  • Database overview: KoNViD-1k, LIVE-VQC, and YouTube-UGC are large-scale UGC-VQA databases built from authentic or in-the-wild video content and human ratings.KoNViD-1k contains 1,200 videos rated by 642 crowd-workers; LIVE-VQC contains 585 videos rated by 4,776 participants; YouTube-UGC contains 1,380 clips rated by more than 8,000 subjects.
  • Content diversity: Content diversity is characterized with brightness, contrast, colorfulness, sharpness, spatial information, and temporal information computed across sampled frames.Each feature is calculated on every tenth frame and averaged over frames for each content.
  • Database overview: LIVE-VQC contains natural mobile-device content, whereas KoNViD-1k and YouTube-UGC also include animations and gaming sources.YouTube-UGC is categorized, while the other two video sets are not.
  • Content diversity: Relative range and uniformity of coverage quantify inter- and intra-database differences in source content characteristics.Uniformity is computed as the entropy of a B-bin feature histogram, with higher values indicating more uniform coverage.
  • Content diversity: The analysis additionally compares content diversity using 4,096-dimensional VGG-19 features embedded into two dimensions with t-SNE, alongside MOS distributions.MOS distributions indicate the quality range represented by each database.

B. Observations

The three UGC-VQA databases differ in their coverage of low-level visual content features, with KoNViD-1k and YouTube-UGC often spanning broader or shifted distributions than LIVE-VQC. Overall range and coverage analyses suggest that large content repositories can produce more diverse and uniformly distributed datasets than direct camera capture.

  • The three databases are compared through feature distributions and paired-space convex hulls across brightness, contrast, colorfulness, sharpness, SI, and TI.
  • KoNViD-1k and YouTube-UGC show similar brightness and contrast coverage, while LIVE-VQC concentrates closer to middle values.
  • KoNViD-1k is skewed toward higher colorfulness, whereas YouTube-UGC has the widest sharpness and spatial-information spread.
  • Overall range and uniformity comparisons indicate that crawling and sampling a large content repository likely yields more diverse, uniformly distributed datasets than sampling directly from user cameras.

III. UGC-VQA MODELS

UGC-VQA requires blind models because consumer videos lack pristine references and contain unpredictable mixtures of authentic impairments. Existing BVQA approaches span handcrafted natural-scene-statistics, codebook, and efficient impairment-oriented models, alongside learning-based methods.

  • Only blind or no-reference VQA models are appropriate when reference videos are unavailable for UGC content.
  • Natural-scene-statistics BVQA models characterize quality through predictable deviations in perceptually relevant low-level features.
  • FRIQUEE achieves state-of-the-art performance on several UGC and consumer databases by combining NSS features from diverse color spaces and transform domains.
  • TLVQM uses two-level extraction of motion, artifact, and aesthetic features for efficient computation, but requires about 30 parameters that may affect transfer to unfamiliar settings.

B. Deep Convolutional Neural Network-Based BVQA Models

Deep CNN-based BVQA models learn spatio-temporal quality representations or combine feature extraction with codec and transfer-learning objectives. The paper motivates fusing complementary model features because performance varies across datasets and perceptual domains.

  • Deep CNNs have been applied to perceptual quality modeling as larger subjective quality databases became available.
  • DeepVQA learns spatio-temporal visual sensitivity maps with a deep ConvNet and convolutional aggregation network.
  • V-MEON jointly trains a 3D-CNN feature extractor and codec classifier in a multi-task framework for quality prediction.
  • Feature-selection performance is evaluated as a function of k, with variability summarized by error bars over repeated iterations.
  • The paper proposes fusing selected features from different BVQA models because their representations may capture distinct perceptual domains and yield more reliable cross-database performance.

A. Feature Extraction

VIDEVAL begins with selected features from high-performing, efficient BVQA models and enriches spatial statistics with temporal pooling. Feature selectors and repeated train-test splits then distill a compact final subset for evaluation.

  • The initial feature set excludes computationally expensive and duplicated features from existing BVQA models.
  • Spatial features are sampled every second frame within one-second chunks and pooled by averages and standard deviations to capture temporal variation.
  • Random forest, linear-kernel SVM, and SFFS are used to rank or search feature subsets, with cross-validation mean squared error helping avoid overfitting.
  • A two-step procedure first selects the feature-selection method and k, then counts feature frequency across 100 random train-test splits to form the final subset.
  • The study compares evaluated blind BVQA models using PLCC, SRCC, and KRCC distributions over repeated random splits.

V. EXPERIMENTAL RESULTS

The evaluation benchmarks representative blind VQA models on three UGC databases and a calibrated combined benchmark using a common SVR-based protocol. It also defines dataset handling, baselines, and performance metrics for comparison.

  • UGC Dataset Benchmarks: 1,323 grayscale-excluded color videos form the YouTube-UGCc subset used to evaluate color models.FRIQUEE and VIDEVAL were evaluated on YouTube-UGCc because grayscale videos caused numerical errors.
  • UGC Dataset Benchmarks: 3,108 videos form the All-Combinedc benchmark by combining KoNViD-1k, LIVE-VQC, and YouTube-UGCc after MOS calibration.YouTube-UGC is the anchor set, and the other datasets’ MOS values are mapped to the scale using INLSA.
  • BVQA Model Benchmarks: 13 representative BVQA and BIQA algorithms are included as reference models, spanning completely blind and training-based approaches.NIQE, ILNIQE, and VIIDEO are opinion-unaware; the remaining listed models are opinion-aware and retrained when evaluated.
  • Visualization: Figure 12 compares model predictions with MOS using scatter plots and nonlinear logistic fitted curves on All-Combinedc.The figure includes hand-designed, deep-feature, temporal, and VIDEVAL models.
  • Regression Models and Metrics: SVR learns feature-to-score mappings, with C and γ optimized through a 10 × 10 exponential grid search.The regression protocol uses randomly split 80%/20% training and test sets, repeated 100 times, with median performance reported.

B. Performance on Individual and Combined Datasets

Completely blind models perform poorly, while VIDEVAL achieves state-of-the-art or near-state-of-the-art results across the evaluated datasets. Performance varies by dataset and model design, with motion and UGC-specific training affecting outcomes.

  • Individual and Combined Datasets: None of NIQE, ILNIQE, and VIIDEO performs very well on the four UGC-VQA benchmarks.The results indicate substantial room for improving opinion-unaware completely blind UGC video quality models.
  • Individual and Combined Datasets: VIDEVAL achieves state-of-the-art or near-state-of-the-art performance on all four test sets among learning-based BVQA models.The comparison uses SVR-trained models under the paper’s common evaluation framework.
  • Dataset-Dependent Performance: TLVQM outperforms other BVQA models on LIVE-VQC but significantly underperforms on YouTube-UGC.The authors relate this pattern to differences in camera motion and TLVQM’s motion-related features.
  • Deep Features: VGG-19 and ResNet-50 perform well on KoNViD-1k and All-Combinedc despite being trained as picture-only models.The authors identify transfer learning as promising for blind UGC-VQA and suggest temporal training could further improve deep models.
  • Deep Features: PaQ-2-PiQ and KonCept512 perform poorly on the three evaluated video datasets.The paper attributes this likely to differences between their spatial picture-quality training distortions and UGC-VQA distortions, motivating UGC-specific training or fine-tuning.

C. Performance Evaluation on Categorical Subsets

The paper evaluates BVQA models across resolution, content, and quality subsets to characterize performance in practical UGC scenarios. VIDEVAL is robust across content and quality categories, while resolution-related effects favor some learned features.

  • Categorical Evaluation: Three categorical evaluation methodologies divide videos by resolution, quality, and content category.These breakdowns are intended to assess model behavior from additional practical perspectives in real-world UGC scenarios.
  • Resolution: ResNet-50 ranks first across the resolution breakdown, while learned features generally outperform hand-designed features.The subsets contain 427 1080p videos, 566 720p videos, and 448 videos at 480p or below.
  • Resolution: Resolution correlates with video quality, with resolution-only features achieving 0.576/0.571 SRCC/PLCC on YouTube-UGC.MSCN distributions also separate the tested downscaled resolutions, suggesting resolution information is embedded in natural-scene statistics.
  • Limitation: Resolution-breakdown evaluation may reduce NSS-based model performance because it removes resolution information that correlates with quality.The paper identifies this as a limitation affecting models such as FRIQUEE and VIDEVAL.
  • Content Categories: VIDEVAL outperforms the other models across Screen Content, Animation, and Gaming categories.The evaluated subsets contain 163 Screen Content, 81 Animation, and 209 Gaming videos.
  • Quality Categories: VIDEVAL remains the best-performing BVQA model on both Low Quality and High Quality partitions.The combined set is split at the median quality value 3.5536 into 1,558 low-quality and 1,550 high-quality videos.

D. Cross Dataset Generalizability

Cross-dataset evaluation tests whether models trained on one UGC database generalize to another after aligning MOS scales. VIDEVAL and ResNet-50 emerge as the most robust models among those compared.

  • Evaluation Protocol: Cross-dataset evaluation trains on one full database and tests on another using SVR and k-fold cross-validation.LIVE-VQC MOS values are linearly scaled from [0, 100] to the scale used by the other datasets.
  • Cross-Dataset Results: Generalization between LIVE-VQC and KoNViD-1k is surprisingly good and is well characterized by pretrained ResNet-50 features.The evaluation compares LIVE-VQC, KoNViD-1k, and YouTube-UGCc.
  • Cross-Dataset Results: Generalization between KoNViD-1k and YouTube-UGC is better than generalization involving LIVE-VQC for the best model, VIDEVAL.The authors relate this to greater overlap in content-space coverage between KoNViD-1k and YouTube-UGC.
  • Cross-Dataset Results: VIDEVAL and ResNet-50 are the most robust BVQA models among those compared for cross-dataset generalization.The conclusion is based on the reported SRCC, PLCC, and cross-dataset evaluations.

E. Effects of Temporal Pooling

The study compares temporal pooling strategies for BVQA predictions and evaluates their reliability, efficiency, and computational cost. Mean-based and ensemble pooling are generally reliable, while low-quality-weighted strategies underperform with few temporal samples.

  • Pooling performance: Mean-family and ensemble pooling were the most reliable strategies on the tested BVQA datasets.Traditional sample mean prediction was often adequate because of its simplicity.
  • Pooling performance: Pooling strategies that emphasize low-quality parts did not perform well, likely because each video provided only 8–20 samples.The limited sample count may prevent statistically meaningful pooling estimates.
  • Evaluation protocol: Chunk-wise prediction enabled temporal pooling but makes Table XI results self-consistent rather than comparable with prior video-wise experiments.The change in prediction granularity may affect base performance.
  • Computational cost: The runtime comparison evaluates feature-based and deep-learning BVQA models on twenty 1080p LIVE-VQC videos, excluding deep-model loading time.Experiments used MATLAB or Python implementations on CPU and GPU hardware described by the authors.
  • Computational cost: VIDEVAL achieves reasonable complexity among top-performing algorithms such as TLVQM and FRIQUEE.The study also reports theoretical time complexity for analytical comparison.
  • Computational cost: TLVQM and VIDEVAL are slower than deep CNN models, while BRISQUE and HIGRADE remain comparatively efficient.The authors expect orders-of-magnitude speedups from pure C/C++ reimplementation of MATLAB models.
  • Computational cost: Regressor training and testing become non-negligible as feature dimensionality grows, especially for codebook-based models such as CORNIA and HOSA.CORNIA uses 10k features and HOSA 14.7k features, substantially more than other handcrafted-feature models.

G. Ensembling VIDEVAL with Deep Features

The authors augment VIDEVAL with deep-learning features and evaluate whether feature fusion improves UGC-VQA. Fusion can improve performance, but the feature dimensions must remain balanced to avoid allowing large deep representations to dominate.

  • Feature fusion: Deep features from PaQ-2-PiQ, KonCept512, VGG-19, and ResNet-50 were appended to VIDEVAL for ensemble fusion.PaQ-2-PiQ contributed local and global quality scores, while KonCept512 contributed its 256-dimensional pre-linear-layer vector.
  • Results: Up to ∼4% performance improvement over vanilla VIDEVAL was obtained by ensembling it with selected deep-learning models.Fusion with ResNet-50 or KonCept512 yielded top performance.
  • Feature fusion: Combining 60-dimensional VIDEVAL features with 4,096-dimensional VGG-19 features may be undesirable because the larger representation could dominate.The authors associate this imbalance with performance drops in Table XIV.
  • Empirical observations: Picture-only models performed remarkably on some Internet UGC videos, whereas motion-related features were more advantageous for mobile-captured videos with frequent camera motion.The reported examples include HIGRADE, FRIQUEE, HOSA, ResNet-50, V-BLIINDS, TLVQM, and VIDEVAL.
  • Empirical observations: Deep CNN descriptors pretrained for classical vision tasks transferred well to UGC video quality prediction.The authors identify transfer learning as a promising direction for general UGC-VQA.
  • Conclusion: The study concludes that selected statistical and visual-impairment features can deliver robust state-of-the-art performance at reasonable computational cost.The authors position VIDEVAL as a compact fusion-based model evaluated within a unified and reproducible framework.
Loading 2005.14354v2…