PaperlayerView exact revision on arXiv

Source-linked AI summary

Blindly Assess Quality of In-the-Wild Videos via Quality-aware Pre-training and Motion Perception

Bowen Li, Weixia Zhang, Meng Tian, Guangtao Zhai, Xianpei Wang

arXiv:2108.08505v2eess.IVcs.CV

TL;DR

Blind VQA must assess videos without pristine references despite diverse authentic distortions and limited annotated video data. The paper transfers quality-aware spatial knowledge from authentic-distortion IQA databases and motion knowledge from action recognition, then trains with mixed list-wise ranking loss; experiments on six databases report competitive individual- and mixed-database performance.

  • Problem

    In-the-wild BVQA lacks accessible pristine references and faces complex authentic distortions, while human-labeled video-quality data are limited for training DNNs.

  • Method

    The method pre-trains frame-level quality features on authentic-distortion IQA databases, transfers motion features from action recognition, fuses them, and fine-tunes with mixed list-wise ranking loss.

  • Results

    The method performs competitively under individual-database and mixed-database training, with ablations supporting quality-aware pre-training, spatial-motion complementarity, and mixed loss.

  • Takeaways & Limitations

    Quality-aware spatial knowledge and transferred motion information provide complementary components for blind assessment of in-the-wild videos.

  • Takeaways & Limitations

    A more rational spatio-temporal interaction strategy remains challenging, alongside incorporating viewing conditions and continual learning for streaming BVQA data.

Abstract

from arXiv · show

Perceptual quality assessment of the videos acquired in the wilds is of vital importance for quality assurance of video services. The inaccessibility of reference videos with pristine quality and the complexity of authentic distortions pose great challenges for this kind of blind video quality assessment (BVQA) task. Although model-based transfer learning is an effective and efficient paradigm for the BVQA task, it remains to be a challenge to explore what and how to bridge the domain shifts for better video representation. In this work, we propose to transfer knowledge from image quality assessment (IQA) databases with authentic distortions and large-scale action recognition with rich motion patterns. We rely on both groups of data to learn the feature extractor. We train the proposed model on the target VQA databases using a mixed list-wise ranking loss function. Extensive experiments on six databases demonstrate that our method performs very competitively under both individual database and mixed database training settings. We also verify the rationality of each component of the proposed method and explore a simple manner for further improvement.

I. INTRODUCTION

Blind video quality assessment is needed for in-the-wild videos because pristine references are often inaccessible and authentic distortions are diverse. The paper addresses these challenges with quality-aware and motion-informed transfer learning, trained using a mixed list-wise ranking loss.

  • Challenges: Reference-based VQA methods are unsuitable when pristine videos are inaccessible or nonexistent, increasing the need for blind VQA.Subjective testing is labor-intensive and time-consuming, while objective VQA predicts quality automatically.
  • Challenges: In-the-wild videos contain authentic distortions from photographing, camera devices, shooting environments, and post-processing, challenging general-purpose BVQA models.Earlier methods mainly targeted specific or synthetic distortions with hand-crafted features.
  • Challenges: Direct DNN-based VQA faces high video-processing cost and insufficient human-annotated quality data for training from scratch.Large-scale pseudo-labeled video pre-training handles synthetic distortions but can remain sub-optimal for authentic distortions.
  • Approach: The proposed method learns quality-aware frame features from IQA databases and motion features from action recognition, then aggregates them for BVQA.A mixed list-wise ranking loss trains the complete model.
  • Approach: The paper combines spatial and temporal information because frame-level quality features and motion information address complementary aspects of video quality.This approach follows a BVQA direction that integrates frame-based quality assessment with motion modeling.

III. PROPOSED METHOD

The proposed method uses transfer learning from quality assessment and action recognition to construct a spatiotemporal representation for blind VQA. It pre-trains frame-level quality features, incorporates motion information, and optimizes the model with a mixed list-wise ranking loss.

  • Overall Framework: The framework first learns a frame-level extractor, then fuses motion information into a spatiotemporal representation, and finally applies mixed list-wise ranking loss.The overall framework is presented in Fig. 1.
  • 1) Transfer Learning:: The method uses pre-training followed by fine-tuning because large video databases with human quality annotations are scarce.Transfer learning is flexible because source and target domains need not share label spaces, although distributional shifts can limit effectiveness.
  • 1) Transfer Learning:: Authentically distorted IQA databases are selected as source domains because their distortions are assumed to better match in-the-wild VQA targets.CORAL is used to compare feature distances between ImageNet, four IQA databases, and target domains.
  • 3) Training Frame-level Feature Extractor:: Quality-aware pre-training models pairwise perceptual-quality probabilities using predicted means and standard deviations from a frame-level network.The network contains a backbone φ and fully connected layers hµ and hσ for estimating the mean and standard deviation.
  • 3) Training Frame-level Feature Extractor:: The training objective combines fidelity and hinge losses, with a regularizer on predicted uncertainty to resolve scaling ambiguity in the standard-deviation estimate.The regularizer receives pairwise labels based on the ordering of ground-truth standard deviations.
  • 3) Training Frame-level Feature Extractor:: Image pairs from four IQA databases are randomly sampled to form the pre-training set, and stochastic gradient descent with L2 weight decay optimizes the network.After training, the backbone extracts frame-level video features.

4) Qualitative Demonstration:

The qualitative demonstration compares representative realistic impairments across source IQA images and target video frames. The examples include blur, grain, exposure, shake, and color distortions.

  • 4) Qualitative Demonstration:: The demonstration visualizes six realistic impairment types: Blurry, Grainy, Underexposed, Shaky, Overexposed, and Poor Color.Each sample is labeled with a single dominant distortion for clearer visualization.

B. Motion Perception

The method combines spatial quality features with motion features to represent in-the-wild videos, then models temporal quality variation with recurrent and hysteresis-based pooling.

  • Motion Perception: Motion information complements spatial appearance because dynamic changes are a distinctive video characteristic and hand-crafted motion features have limited efficiency or representational power.The method therefore transfers motion-related knowledge from a pre-trained 3D action-recognition network.
  • Spatial Features: Quality-aware pre-training trains a ResNet-50 frame extractor, whose final convolutional activations are aggregated with global average and standard deviation pooling.The resulting spatial representation concatenates GAP and GSP features for each frame.
  • Motion Features: SlowFastF captures motion and produces pooled 512-dimensional frame-level features from video clips.Its default configuration uses temporal stride τ = 8, speed ratio α = 4, and channel ratio β = 1/8.
  • Feature Fusion: Spatial and motion features are temporally aligned, concatenated, and converted into 4,608-dimensional frame-level vectors with temporal length T/2.The spatial tensor is sampled every two frames to match the motion pipeline.
  • Temporal Modeling and Quality Prediction: A GRU models temporal information, maps hidden states to frame-level quality scores, and feeds them into temporal hysteresis pooling for video-level prediction.Hysteresis pooling combines memory quality based on prior worst cases with a current quality item emphasizing rapid quality drops, then averages time-varying scores.

E. Loss Function

The loss function combines differentiable PLCC and SRCC objectives to encourage both prediction precision and monotonic agreement with subjective quality ratings.

  • PLCC Loss: PLCC loss optimizes prediction precision after applying a nonlinear logistic mapping to the model predictions.The mapping is implemented as a network module using Linear, Sigmoid, and Linear layers.
  • SRCC Loss: SRCC loss provides a differentiable proxy for prediction monotonicity by computing correlation between soft ranks of predictions and ground-truth quality.The approach addresses the non-differentiability of conventional ranking operations.
  • Mixed Loss: The overall loss combines SRCC and PLCC losses with a trade-off parameter λ.Both components are list-wise ranking losses applicable to individual- or mixed-database training.

IV. EXPERIMENTS

The experiments benchmark the method across individual, mixed, and cross-database scenarios, then assess its components through qualitative examples, ablation, and computational analysis.

  • IV. EXPERIMENTS: The evaluation covers individual, mixed, and cross-database training scenarios.The experimental section also includes qualitative results, ablation studies, and computational complexity analysis.

1) Benchmarking Databases:

The study evaluates blind video quality assessment on six heterogeneous in-the-wild databases using correlation-based criteria and comparisons with adapted image- and video-quality models.

  • Benchmarking Databases: Six databases—CVD2014, KoNViD-1k, LIVE-Qualcomm, LIVE-VQC, YouTube-UGC, and LSVQ—differ in content, resolution, duration, and annotation scale.The latest YouTube-UGC version contributes 1,142 videos after excluding 57 grayscale videos, while LSVQ has 39,072 MOS ground truths.
  • Reported Results: Tables III and IV report individual-database SRCC and PLCC results for five databases and LSVQ, respectively.Table III reports median results with standard deviations, while Table IV shows database size in brackets.
  • Competing Methods: The benchmarks compare the proposed method with adapted BIQA models and representative BVQA methods.Adapted BIQA baselines extract frame-level features and apply temporal average pooling to obtain video-level representations.
  • Evaluation Protocol: SRCC measures prediction monotonicity and PLCC measures prediction accuracy after four-parameter logistic mapping.Except for LSVQ, databases are split into 60% training, 20% validation, and 20% testing without content overlap.

4) Implementation Details:

The method is evaluated under individual, mixed-database, and category-level settings. Results show strong performance across databases and robustness across resolutions, contents, and quality levels.

  • Individual Database Performance: The proposed method achieves superior performance on all in-the-wild VQA databases, including a +12.38% SRCC margin over VSFA on LIVE-VQC.It also performs strongly on KoNViD-1k, YouTube-UGC, and LSVQ.
  • Database-level Mixed Test: Mixed list-wise ranking consistently outperforms MDTVSFA across the mixed-database evaluation.It also surpasses linear rescaling with L1 regression in overall performance across all databases.
  • Database-level Mixed Test: Mixed-database training improves performance on CVD2014 and LIVE-Qualcomm while causing slight drops on KoNViD-1k and LIVE-VQC.The authors associate this pattern with reduced overfitting on small databases and acceptable disturbance on larger ones.
  • Category-level Mixed Test: The category-level evaluation combines KoNViD-1k, LIVE-VQC, and YouTube-UGC into calibrated mixed subsets for further analysis.The subsets cover resolution, content, and quality categories.
  • Category-level Mixed Test: The proposed method is effective and robust across different resolutions, contents, and quality levels.The results further support quality-aware pre-training and motion perception over image-classification pre-training for varied contents and quality levels.

D. Cross-database Evaluation

The proposed method generalizes well across unseen VQA databases, while ablations show that quality-aware spatial features, motion features, mixed loss, and ensemble prediction each contribute to performance. The qualitative analysis also reveals difficulty discriminating among relatively high-quality videos.

  • Cross-database Evaluation: Over 9% SRCC improvement is achieved on small and medium databases, while LSVQ gains more than 4% SRCC in cross-database evaluation.These results indicate favorable generalizability to unseen databases.
  • Qualitative Results: The model preserves quality ordering in successful samples but makes ranking errors among relatively high-quality videos in failure cases.The reported failure comparisons remain reasonable in absolute quality prediction despite disagreement with MOS rankings.
  • Feature Ablation: 4.23% higher weighted average SRCC comes from quality-aware spatial features alone compared with the ImageNet-pretrained baseline.Motion features alone reduce performance, but combining spatial and motion features produces the strongest results.
  • Feature Ablation: +6.6% SRCC from motion features is observed on LIVE-VQC, where motion-related distortions are prevalent.This supports the complementarity of motion and spatial information for video quality assessment.
  • Loss Ablation: The mixed SRCC-and-PLCC loss adds 0.93% weighted SRCC over using either loss alone and improves several databases.The mixed loss performs particularly well on CVD2014, LIVE-Qualcomm, and YouTube-UGC.
  • Ensemble Ablation: A GRU slightly outperforms a Transformer, while combining their predictions adds 0.92% weighted average SRCC.The ensemble combines the two predicted scores using a factor κ varied from 0 to 1.

G. Computational Complexity

The study benchmarks runtime and prediction performance across implementations, fixed video conditions, and multiple resolutions. GPU execution provides substantial acceleration, especially as resolution increases, while CPU execution is more computationally demanding.

  • Computational Complexity: Figure 6 plots weighted average SRCC against logarithmic runtime and runtime against video resolution.The resolution analysis fixes a 1280×720 video for the runtime-performance comparison and varies resolution from 360p to 2160p for the second plot.
  • Computational Complexity: 3 to 16 times faster than TLVQM is achieved by Proposed GPU PAL as resolution increases from 360p to 2160p.Runtime is evaluated across six resolutions using ten repeated tests.
  • Computational Complexity: Higher computational complexity on CPU contrasts with better prediction accuracy, while GPU execution provides significant acceleration.The comparison uses the same workstation and author-released implementations for all methods.

V. CONCLUSION

The paper presents a DNN-based BVQA method that transfers knowledge from image-quality and motion domains to represent in-the-wild videos. It reports performance gains from complementary spatial-temporal modeling and identifies unresolved interaction, viewing-condition, and streaming-data challenges.

  • Conclusion: Knowledge from multiple IQA databases and action recognition is transferred to learn spatial appearance and temporal motion representations for in-the-wild BVQA.The model combines quality-aware frame-level pre-training with motion information and a mixed loss.
  • Conclusion: A more rational spatio-temporal interaction strategy remains a challenging limitation of the current model.The paper also identifies viewing conditions and continual learning for streaming BVQA as future directions.
Loading 2108.08505v2…