Source-linked AI summary
RAPIQUE: Rapid and Accurate Video Quality Prediction of User Generated Content
Zhengzhong Tu, Xiangxu Yu, Yilin Wang, Neil Birkbeck, Balu Adsumilli, Alan C. Bovik
TL;DR
UGC quality prediction lacks accurate, efficient no-reference models because authentic videos contain complex distortions and no pristine reference. RAPIQUE combines spatial and temporal NSS with deep CNN features, achieving strong performance across UGC databases while substantially reducing runtime. Its design supports practical video-quality modeling, though the supplied passages state no specific limitation.
Problem
No-reference UGC video quality assessment must handle complex, diverse distortions without pristine references while remaining accurate and efficient for compression and transcoding applications.
Method
RAPIQUE combines spatial and temporal NSS features with high-level CNN features in a two-branch framework and jointly trains a regressor head.
Results
RAPIQUE delivers uniformly strong performance across four UGC test sets, including best performance on KoNViD-1k and All-Combined, with 20x speedup on 1080p videos and 60x faster runtime than VIDEVAL on 4k videos.
Takeaways & Limitations
RAPIQUE provides an efficient, database-agnostic BVQA model whose spatial and temporal statistics can support practical UGC quality prediction.
Abstract
from arXiv · showhide
Blind or no-reference video quality assessment of user-generated content (UGC) has become a trending, challenging, heretofore unsolved problem. Accurate and efficient video quality predictors suitable for this content are thus in great demand to achieve more intelligent analysis and processing of UGC videos. Previous studies have shown that natural scene statistics and deep learning features are both sufficient to capture spatial distortions, which contribute to a significant aspect of UGC video quality issues. However, these models are either incapable or inefficient for predicting the quality of complex and diverse UGC videos in practical applications. Here we introduce an effective and efficient video quality model for UGC content, which we dub the Rapid and Accurate Video Quality Evaluator (RAPIQUE), which we show performs comparably to state-of-the-art (SOTA) models but with orders-of-magnitude faster runtime. RAPIQUE combines and leverages the advantages of both quality-aware scene statistics features and semantics-aware deep convolutional features, allowing us to design the first general and efficient spatial and temporal (space-time) bandpass statistics model for video quality modeling. Our experimental results on recent large-scale UGC video quality databases show that RAPIQUE delivers top performances on all the datasets at a considerably lower computational expense. We hope this work promotes and inspires further efforts towards practical modeling of video quality problems for potential real-time and low-latency applications. To promote public usage, an implementation of RAPIQUE has been made freely available online: \url{https://github.com/vztu/RAPIQUE}.
I. INTRODUCTION
UGC video quality assessment is difficult because authentic videos contain diverse, commingled distortions without pristine references, while existing models trade accuracy against computational efficiency. RAPIQUE addresses this gap by combining spatial and temporal NSS with deep features in an efficient framework.
- Motivation: Authentically captured UGC videos contain diverse, commingled distortions and lack pristine references, making quality prediction harder than synthetic-distortion assessment.
- Motivation: Accurate and efficient no-reference quality prediction can guide compression settings and adaptive transcoding profiles for UGC pipelines.
- Prior limitations: Existing BVQA models either miss complex UGC distortions or require expensive motion estimation and complicated statistical features.
- RAPIQUE: RAPIQUE combines spatial and temporal NSS features with CNN features, then jointly learns a regressor for video quality prediction.
- RAPIQUE: RAPIQUE achieves comparable or better SOTA performance with a relative 20x speedup on 1080p videos and 60x faster runtime than VIDEVAL on 4k videos.
- RAPIQUE: Its temporal model uses bandpass regularities beyond frame differences, while its spatial NSS module offers an efficient alternative to FRIQUEE.
B. Deep Learning-Based BVQA Models
Deep CNNs have improved perceptual quality modeling, but models trained on synthetic distortions often struggle on in-the-wild UGC databases. Representative approaches use pretrained or augmented CNNs and specialized video architectures.
- Deep learning motivation: Deep CNNs have delivered strong performance in low-level vision and became more applicable to quality modeling as large-scale psychometric databases emerged.
- Training strategies: Researchers address limited quality-data size through patch-wise augmentation or ImageNet pretraining followed by fine-tuning on target databases.
- Limitation: CNN-based models can perform remarkably on synthetic-distortion datasets yet struggle on recent in-the-wild UGC databases.
- Representative models: DeepVQA learns spatio-temporal visual sensitivity maps with a deep CNN and convolutional aggregation network.
- Representative models: V-MEON jointly optimizes a 3D-CNN feature extractor and codec classifier before predicting video quality with fully connected layers.
III. RAPID AND ACCURATE VIDEO QUALITY EVALUATOR (RAPIQUE)
RAPIQUE combines efficient natural-scene-statistics features with deep features in a two-branch quality-prediction framework. Its NSS module models spatial and temporal regularities whose distortions are informative for UGC quality.
- RAPIQUE combines low-level scene statistics with high-level deep-learning features in an efficient two-branch framework.The framework uses spatial and temporal sampling to exploit content and distortion redundancies.
- Natural Scene Statistics: MSCN coefficients of distorted images deviate from characteristic statistical regularities, making those deviations useful quality features.The coefficients are modeled with a zero-mean generalized Gaussian distribution using shape and spread parameters.
- Feature Maps: The extracted feature maps include luma, gradient, multiscale bandpass, and chromatic representations for quality modeling.The maps include Y, GM, LoG, DoG, opponent-color components, and CIELAB chroma maps.
- Natural Scene Statistics: Four-direction MSCN pair products capture distortion-related correlations, while seven log-derivative statistics exhibit distinct quality-dependent deviations.Pair products are modeled with asymmetric generalized Gaussian distributions, and log-derivative parameters provide additional learning features.
B. Spatial Features
RAPIQUE constructs spatial features from multiscale luma and chromatic maps using a compact NSS-34 operator. This design captures structure, color, and cross-scale quality cues while remaining substantially faster than FRIQUEE.
- NSS-34: The NSS-34 module extracts 34 features from each input image or feature map using MSCN, variance-field, AGGD, and GGD statistics.These features are computed from MSCN transforms, variance fields, four directional pair products, and seven paired log-derivative directions.
- Bandpass Maps: GM captures orthogonal directional gradients, while LoG and DoG provide high- and mid-frequency bandpass responses for frame structures.DoG is formed from Gaussian filters with different standard deviations, and only its first decomposition level is used.
- Chromatic Features: The chromatic feature set includes opponent-color, CIELAB chroma, and corresponding gradient maps alongside luma representations.RAPIQUE uses BY, RG, A, and B chromatic maps and their gradient maps.
- Multiscale Spatial Features: RAPIQUE applies NSS-34 at two scales to four luma maps and at half scale to twelve chroma maps, yielding 680 spatial features.The luma maps are Y, GM, LoG, and DoG; reduced chroma processing reflects YUV420 representation and lower chroma sensitivity.
- Efficiency: The 680-dimensional spatial model achieves comparable performance to FRIQUEE while running 20x faster.The comparison is against FRIQUEE’s 560-dimensional model.
C. Temporal Features
RAPIQUE models temporal distortions with bandpass regularities of natural videos rather than relying only on motion estimation or simple frame differences. It analyzes decorrelated temporal subband statistics to derive temporal quality features.
- RAPIQUE introduces a temporal statistics model based on bandpass regularities of natural videos, going beyond simpler frame-difference models.
- Temporal bandpass responses are computed by convolving the video with a bank of one-dimensional temporal filters.Frame differences are a special case of this formulation, corresponding to the high-pass component of a 2-tap Haar wavelet.
- The model applies MSCN transforms to seven temporal bandpass subbands while sampling frame times non-densely.
- The MSCN coefficients exhibit Gaussianlike distributions in natural videos, whose distortion-related deviations support perceived-quality prediction.
- Two-scale processing of seven subbands produces a 476-dimensional temporal feature vector from 34 features per band.The feature count is (34 features/band)×(7 subbands)×(2 scales) = 476.
- Spatial features are average- and absolute-difference-pooled within non-overlapping one-second chunks to capture feature variation over time.
D. Deep Learning Features
RAPIQUE supplements low-level NSS features with efficient high-level CNN features because CNN-based methods have not generally proven effective on UGC video quality databases. It uses aggressively downscaled inputs and a pretrained ResNet-50 feature extractor.
- CNN-based solutions perform well on UGC picture quality problems, but none had proven effective on the cited UGC video quality databases.
- RAPIQUE combines low-level quality-aware scene statistics with high-level semantic CNN features.
- The CNN component is included to capture high-level semantic information while supplementing, rather than replacing, spatial and temporal NSS features.
- RAPIQUE aggressively downscales frames for CNN processing to improve efficiency relative to previous CNN-based video quality models.
- A pretrained ResNet-50 provides a 2,048-dimensional semantic feature vector without fine-tuning.The paper gives overfitting prevention as an additional reason for using a pretrained CNN without fine-tuning.
E. Learning a Video Quality Predictor
RAPIQUE improves speed through aggressive spatial and temporal sampling while extracting pooled spatial, temporal-variation, and temporal-bandpass features from sampled video frames.
- RAPIQUE uniformly samples two frames per second to extract 680-dimensional spatial NSS features.
- Average pooling and absolute-difference pooling convert the sampled spatial features into 680 spatial and 680 temporal-variation features.
- RAPIQUE samples eight consecutive frames each second and applies a temporal Haar filter to obtain seven bandpass responses.
- The feature computations use a higher resized resolution while maintaining aspect ratio, with 512p used in the experiments.
A. Experiment Settings
Experiments evaluate BVQA models on three recent datasets and their calibrated composite union using SVR-based regression and repeated train-test splits. PLCC and RMSE are computed after nonlinear logistic regression.
- Datasets and Baselines: The evaluation uses KoNViD-1k, LIVE-VQC, YouTube-UGC, and the composite All-Combined benchmark.All-Combined is the union of KoNViD-1k (1,200), LIVE-VQC (575), and YouTube-UGC (1,380) after MOS calibration.
- Datasets and Baselines: KoNViD-1k and LIVE-VQC are calibrated with equations (20) and (21), respectively, while YouTube-UGC is unchanged.
- Datasets and Baselines: The compared baselines include spatial-only models, feature-based BVQA models, and deep learning models.
- Evaluation Method: An SVR learns feature-to-score mappings, with C and γ optimized by randomized grid search on the training set.
- Evaluation Method: Datasets are randomly split 80%/20% into training and test sets over 20 iterations, and median test performance is reported.
- Evaluation Method: PLCC and RMSE are computed after nonlinear four-parametric logistic regression places objective predictions on the MOS scale.
B. Main Evaluation Results
RAPIQUE performs strongly across UGC benchmarks, achieving top or near-top correlations while remaining database-agnostic.
- RAPIQUE achieved the best performance on KoNViD-1k, outperforming recent dense deep learning models including VSFA and MDVSFA.
- On LIVE-VQC, RAPIQUE ranked second behind TLVQM, whose motion-relevant features suit videos with large camera motions.
- RAPIQUE ranked second on YouTube-UGC, slightly behind VIDEVAL, and best on the 3165-video All-Combined set.
- RAPIQUE is database-agnostic and exhibited uniformly strong performance across all four test sets.
C. Effects of Training Data Size
RAPIQUE benefits from additional training data and from combining temporal, deep, and spatial quality features.
- RAPIQUE achieved better than 0.8 PLCC using only 50% of the All-Combined training data.
- As the training-data fraction increased, RAPIQUE improved relative to VIDEVAL and TLVQM, although it did not outperform VIDEVAL below 40%.
- Including deep learning features significantly boosted performance over NSS-only features across the evaluated UGC datasets.
- CNN features provided greater benefits for screen-content and gaming videos than for natural videos.
- The new temporal statistical features produced noticeable improvements over using spatial features alone.
E. Performance on different deep features
RAPIQUE’s ResNet-50 configuration provides strong benchmark performance while retaining the model’s computational efficiency and resolution scalability.
- Combining NSS features with ResNet-50 yielded the best or top performance on all evaluated benchmarks.
- RAPIQUE is 2-150x faster than baseline methods depending on resolution, with speed increasing at higher resolutions.
- ResNet-50 performed slightly better than KonCept512, indicating value from classification-pretrained high-level semantic features.
- Features pretrained on a specific IQA dataset may limit generalizability to future unseen distortions.
- RAPIQUE achieves comparable prediction accuracy with 20x less computational expense than VIDEVAL.
- RAPIQUE combines spatial-temporal scene statistics with deep CNN features in a two-branch framework for video quality prediction.