Source-linked AI summary
A Deep Learning based No-reference Quality Assessment Model for UGC Videos
Wei Sun, Xiongkuo Min, Wei Lu, Guangtao Zhai
TL;DR
UGC VQA lacks a reference video and must handle diverse authentic distortions, while prior methods often use mismatched pretrained image features. The paper learns spatial quality features end to end, adds motion features, and uses efficient multi-scale processing; it reports the best performance on five popular UGC VQA databases. Its generalization is limited because individual databases do not cover all video types and distortions.
Problem
UGC videos lack pristine references and contain diverse authentic distortions, while prior methods commonly rely on image-recognition or pretrained IQA features.
Method
The model learns spatial quality-aware features end to end, extracts complementary motion features, regresses chunk scores with an MLP, temporally pools them, and fuses multiple resolutions using HVS-based weights.
Results
The model achieves the best performance on five popular UGC VQA databases.
Takeaways & Limitations
The proposed framework combines quality-aware spatial and motion representations with low-complexity processing for UGC video quality assessment.
Takeaways & Limitations
A single database cannot cover all video types and distortions, limiting generalization; multiple-database training or continual learning may improve it.
Abstract
from arXiv · showhide
Quality assessment for User Generated Content (UGC) videos plays an important role in ensuring the viewing experience of end-users. Previous UGC video quality assessment (VQA) studies either use the image recognition model or the image quality assessment (IQA) models to extract frame-level features of UGC videos for quality regression, which are regarded as the sub-optimal solutions because of the domain shifts between these tasks and the UGC VQA task. In this paper, we propose a very simple but effective UGC VQA model, which tries to address this problem by training an end-to-end spatial feature extraction network to directly learn the quality-aware spatial feature representation from raw pixels of the video frames. We also extract the motion features to measure the temporal-related distortions that the spatial features cannot model. The proposed model utilizes very sparse frames to extract spatial features and dense frames (i.e. the video chunk) with a very low spatial resolution to extract motion features, which thereby has low computational complexity. With the better quality-aware features, we only use the simple multilayer perception layer (MLP) network to regress them into the chunk-level quality scores, and then the temporal average pooling strategy is adopted to obtain the video-level quality score. We further introduce a multi-scale quality fusion strategy to solve the problem of VQA across different spatial resolutions, where the multi-scale weights are obtained from the contrast sensitivity function of the human visual system. The experimental results show that the proposed model achieves the best performance on five popular UGC VQA databases, which demonstrates the effectiveness of the proposed model. The code will be publicly available.
1 INTRODUCTION
UGC videos exhibit diverse authentic distortions and content, while prior VQA methods often rely on mismatched pretrained image features. The paper proposes end-to-end spatial learning, complementary motion features, and multi-scale fusion for efficient UGC VQA.
- UGC videos vary widely in visual quality because they are captured by consumers with diverse devices and expertise.
- Authentic UGC distortions include exposure problems, low visibility, jitter, noise, and color shifts that require stronger feature representations.
- Prior deep UGC VQA methods commonly extract frame features using image-recognition or pretrained IQA models rather than end-to-end quality learning.
- The proposed model learns spatial quality-aware features from raw frames end to end and extracts motion features to represent temporal distortions.
- The model regresses features with an MLP, pools chunk scores temporally, and fuses quality across resolutions using HVS contrast-sensitivity weights.
- The model is reported to achieve strong performance on five UGC VQA databases while maintaining low computational complexity.
2 RELATED WORK
Related work spans handcrafted spatio-temporal statistics, semantic deep features, and hybrid spatial-temporal networks for no-reference video quality assessment.
- Handcrafted NR VQA methods pool framewise IQA scores or model motion, temporal statistics, and distortion-related features.
- The proposed architecture is organized around feature extraction, quality regression, and quality pooling modules.
- Deep NR VQA methods use multitask learning, pretrained semantic features with recurrent temporal modeling, and multi-database training.
- Other approaches combine multi-scale motion information, 2D spatial features with 3D spatial-temporal features, or compression, content, and distortion cues.
3 PROPOSED MODEL
The proposed NR UGC VQA framework extracts spatial and spatial-temporal quality-aware features, maps them to chunk scores, and pools those scores into a video quality estimate.
- The feature extraction module evaluates spatial and motion distortions using spatial-domain and spatial-temporal quality-aware features.
- The regression module maps extracted features to chunk-level quality scores, while the pooling module produces the video-level quality score.
3.1 Feature Extraction Module
The feature extractor separates spatial and motion distortions, using sparse high-resolution spatial processing and dense low-resolution motion processing to reduce computation.
- Spatial distortions affect frames, whereas motion distortions arise temporally from effects such as jitter, lagging, unstable equipment, or limited bandwidth.
- Spatial features are resolution-sensitive but frame-rate-insensitive, while motion features show the opposite sensitivity pattern.
- Each video chunk contributes one key frame for spatial extraction, while all chunk frames support low-resolution motion extraction.
- The sparse spatial and low-resolution dense motion design greatly reduces computational complexity with little reported performance degradation.
- The spatial CNN learns quality-aware representations from trainable frame features, combining information from its feature-map stages.
- Global means and standard deviations of CNN feature maps are concatenated to form the spatial feature representation.
- A pretrained action-recognition network extracts motion features, which are concatenated with spatial features for each video chunk.
3.2 Quality Regression Module
The quality regression module uses a simple MLP to map quality-aware features to chunk-level video-quality scores.
- 3.2 Quality Regression Module: The MLP maps extracted quality-aware feature representations into chunk-level quality scores.It consists of two fully connected layers with 128 and 1 neuron, respectively.
3.3 Quality Pooling Module
The quality pooling module aggregates chunk-level scores into a video-level quality score using temporal average pooling.
- 3.3 Quality Pooling Module: Each video is split into N_c continuous chunks, and the model obtains one quality score q_i for each chunk.The chunk-level scores are produced by the feature extraction and quality regression modules.
- 3.3 Quality Pooling Module: Temporal average pooling is selected because it achieves the best performance among the evaluated temporal pooling methods.The resulting video-level quality Q is calculated from the chunk-level scores.
3.4 Loss Function
The loss function combines MAE regression with a rank loss to improve both score accuracy and relative-quality discrimination.
- 3.4 Loss Function: The MAE loss brings evaluated quality scores close to the ground-truth scores.The ground-truth score is defined for each video in a mini-batch.
- 3.4 Loss Function: The rank loss helps the model distinguish the relative quality of videos with similar quality.A differentiable formula approximates the non-differentiable rank value between two videos.
- 3.4 Loss Function: A hyper-parameter λ balances the MAE loss and the rank loss in the final objective.The combined loss is formed after calculating both component losses.
3.5 Multi-scale Quality Fusion Strategy
The multi-scale fusion strategy evaluates videos at three resolutions and combines their quality scores using weights derived from human visual sensitivity and viewing conditions.
- 3.5 Multi-scale Quality Fusion Strategy: The strategy addresses quality assessment across different spatial resolutions by fusing quality scores from multiple scales.The authors motivate this design because prior approaches use the original or one fixed resized resolution.
- 3.5 Multi-scale Quality Fusion Strategy: The model evaluates each video at fixed 540p, 720p, and 1080p scales.The design includes up-sampling artifacts that can affect perceived quality when playback devices have higher resolution.
- 3.5.2 Adaptive Multi-scale Weights.: The scale weights are based on the contrast sensitivity function of the human visual system and viewing-environment information.The viewing resolution factor depends on viewing distance, screen height, and vertical pixel count.
- 3.5.2 Adaptive Multi-scale Weights.: Each scale weight is calculated as the area under the CSF over its corresponding spatial-frequency range, then used to compute the fused quality score.The three scales correspond to different frequency-covering ranges, with normalization applied to the weights.
4 EXPERIMENTAL VALIDATION
The model is evaluated across five UGC VQA databases using standard correlation metrics and comparisons with IQA and VQA baselines. It achieves the best performance across all five databases, while multi-scale fusion improves high-resolution evaluation.
- Experimental databases: The experiments evaluate KoNViD-1k, YouTube-UGC, LSVQ, LBVD, and LIVE-YT-Gaming.These databases cover diverse categories, live broadcasting, transmission distortions, and gaming content.
- Evaluation protocol: Performance is measured using PLCC for prediction linearity and SRCC for prediction monotonicity.Values closer to 1 indicate better VQA performance.
- Overall results: The proposed model achieves the best performance on all five UGC VQA databases and leads by a large margin.This result indicates strong quality measurement across varied UGC videos.
- Resolution robustness: Multi-scale quality weighting fusion significantly improves performance on the LSVQ test-1080p subset.Without fusion, the model is inferior to Li et al. on this subset, partly because test videos exceed 1080p while inputs are resized to 448×448.
- Baseline comparison: Handcrafted IQA models perform poorly on UGC databases, especially LBVD and LIVE-YT-Gaming, because their design targets natural images with synthetic distortions.Fine-tuning ResNet50 achieves comparatively good performance, highlighting the importance of spatial features.
4.3 Ablation Studies
Ablation studies examine feature extraction, regression, temporal pooling, cross-database generalization, and computational complexity. Results favor combining spatial and motion features, end-to-end spatial training, and sparse low-resolution processing.
- Ablation design: Ablation studies are conducted on LSVQ because it is the largest and more representative UGC VQA database.The study examines the feature extraction and quality regression modules.
- Feature extraction: Combining spatial and motion features outperforms either feature type alone, while spatial features contribute more strongly.End-to-end training further improves the spatial feature extractor's evaluation performance.
- Quality regression: The proposed MLP regression setup is compared with GRU and Transformer alternatives while keeping other experimental settings unchanged.The supplied passage introduces this comparison but does not report its final numerical outcome.
- Temporal pooling: Average temporal pooling performs similarly to learning-based pooling and better than the subjectively inspired method.The comparison uses a 1D CNN as the learning-based pooling alternative.
- Generalization: Cross-database evaluation trains on LSVQ and tests on four other UGC VQA databases, with SRCC exceeding 0.8 on KoNViD-1k and YouTube-UGC.Multi-scale fusion significantly improves databases containing videos with different spatial resolutions.
- Generalization: Generalization is weaker on LBVD and LIVE-YT-Gaming because live broadcasting and gaming videos may be rare in LSVQ.The authors suggest multiple-database training or continual learning as possible improvement strategies.
- Computational complexity: The proposed model has considerably low running time because spatial features use sparse frames and motion features use very low resolution.The authors therefore describe it as suitable for practical applications.
5 CONCLUSION
The paper presents an effective and efficient no-reference UGC VQA model that combines spatial and motion quality features. It uses MLP regression, temporal average pooling, and multi-scale fusion to assess videos across resolutions.
- Conclusion: The model extracts spatial and motion quality-aware features to measure spatial and motion distortions, respectively.The spatial extractor is trained end-to-end to use varied UGC content and distortions.
- Conclusion: MLP regression and temporal average pooling convert quality-aware features into video-level quality scores.Multi-scale quality fusion addresses quality assessment across different spatial resolutions.
- Conclusion: Experimental results show that the model can effectively measure the quality of UGC videos.