PaperlayerView exact revision on arXiv

Source-linked AI summary

Quality Assessment of In-the-Wild Videos

Dingquan Li, Tingting Jiang, Ming Jiang

arXiv:1908.00375v3cs.MMcs.CVeess.IV

TL;DR

In-the-wild video quality assessment lacks reference videos and faces unknown shooting distortions. The proposed no-reference method integrates human visual-system effects through content-aware features and temporal-memory modeling, outperforming state-of-the-art methods across three databases, while remaining below 0.76 correlation on two databases.

  • Problem

    In-the-wild video quality assessment is challenging because reference videos are unavailable and shooting distortions are unknown.

  • Method

    The method integrates human visual-system content-dependency and temporal-memory effects using content-aware perceptual features, GRU-based long-term dependency modeling, and temporal pooling.

  • Results

    30.21%, 8.63%, and 17.96% SROCC improvements are achieved on LIVE-Qualcomm, KoNViD-1k, and CVD2014, respectively, over five state-of-the-art methods.

  • Takeaways & Limitations

    Content-aware perceptual features and temporal-memory modeling are important for in-the-wild video quality assessment.

  • Takeaways & Limitations

    The temporal pooling model requires reliable frame quality scores and is not differentiable, limiting its direct application to no-reference in-the-wild assessment.

Abstract

from arXiv · show

Quality assessment of in-the-wild videos is a challenging problem because of the absence of reference videos and shooting distortions. Knowledge of the human visual system can help establish methods for objective quality assessment of in-the-wild videos. In this work, we show two eminent effects of the human visual system, namely, content-dependency and temporal-memory effects, could be used for this purpose. We propose an objective no-reference video quality assessment method by integrating both effects into a deep neural network. For content-dependency, we extract features from a pre-trained image classification neural network for its inherent content-aware property. For temporal-memory effects, long-term dependencies, especially the temporal hysteresis, are integrated into the network with a gated recurrent unit and a subjectively-inspired temporal pooling layer. To validate the performance of our method, experiments are conducted on three publicly available in-the-wild video quality assessment databases: KoNViD-1k, CVD2014, and LIVE-Qualcomm, respectively. Experimental results demonstrate that our proposed method outperforms five state-of-the-art methods by a large margin, specifically, 12.39%, 15.71%, 15.45%, and 18.09% overall performance improvements over the second-best method VBLIINDS, in terms of SROCC, KROCC, PLCC and RMSE, respectively. Moreover, the ablation study verifies the crucial role of both the content-aware features and the modeling of temporal-memory effects. The PyTorch implementation of our method is released at https://github.com/lidq92/VSFA.

1 INTRODUCTION

In-the-wild video quality assessment lacks reference videos and must handle complex, temporally heterogeneous real-world distortions. The proposed no-reference method incorporates content-dependency and temporal-memory effects from the human visual system into a deep neural network.

  • Motivation: In-the-wild videos contain unknown shooting distortions and complex mixed real-world artifacts, making quality assessment without pristine references challenging.The paper motivates quality assessment for identifying, monitoring, repairing, or enhancing low-quality videos.
  • Human Visual System Effects: Human quality judgments depend on video content, motivating content-aware features for objective assessment.The paper reports cross-content pairwise comparisons in which more than 7 of 10 subjects preferred one image in 82 of 201 pairs.
  • Human Visual System Effects: Temporal-memory effects make current judgments depend on previous frames, including lower perceived quality after earlier poor-quality frames.This temporal hysteresis means simple average pooling can overestimate quality for videos with fluctuating frame-wise quality.
  • Proposed Method: The method extracts content-aware features with a pre-trained image-classification CNN, models long-term dependencies with a GRU, and uses subjectively inspired temporal pooling.These components integrate content-dependency and temporal-memory effects into a no-reference VQA network.
  • Validation: Experiments on KoNViD-1k, LIVE-Qualcomm, and CVD2014 show superior performance over five state-of-the-art methods.An ablation study verifies the key role of the method’s components and suggests HVS knowledge can improve consistency with human perception.

2 RELATED WORK

Prior video quality assessment methods largely target synthetic distortions, references, or compression artifacts, leaving in-the-wild videos poorly served. This work addresses that gap with content-aware features, GRU-based temporal modeling, and differentiable hysteresis-aware pooling.

  • Video Quality Assessment: Traditional and deep learning-based VQA methods mainly address synthetic distortions, while streaming video quality-of-experience is outside this paper’s scope.Existing approaches use handcrafted statistics, CNNs, or 3D-CNNs, but the deep models described are trained for synthetic or compressed-video settings.
  • Video Quality Assessment: In-the-wild VQA requires no-reference methods because pristine reference videos are unavailable, yet existing no-reference methods perform poorly on these databases.Existing deep models are also unsuitable when they require reference information or focus only on compression artifacts.
  • Content-Aware Features: Content-aware features incorporate semantic information from pre-trained image-classification networks to address content-dependency in quality prediction.Prior work combines content-relevant or deep semantic features with quality measures; this paper is inspired by those approaches.
  • Content-Aware Features: The proposed framework uses a pre-trained CNN with global pooling, followed by dimensionality reduction and temporal-memory modeling.The framework diagram identifies content-aware feature extraction and temporal-memory effects as its two main modules.
  • Temporal Modeling: Unlike temporal averaging, the method uses a GRU to integrate frame features and learn long-term dependencies.The GRU also predicts frame-wise quality scores.
  • Temporal Modeling: The method introduces differentiable subjectively inspired temporal pooling to model hysteresis while allowing neural-network training by backpropagation.The earlier hysteresis pooling model required reliable frame scores and was not differentiable, limiting direct use for this task.

3 THE PROPOSED METHOD

The proposed no-reference video quality assessment method integrates content-aware features with global and local modeling of temporal-memory effects. It uses CNN features, a GRU, and differentiable temporal pooling to predict overall video quality.

  • Motivation: The method is motivated by content dependency and temporal hysteresis, in which poor preceding frames can lower perceived quality after quality recovers.These effects make simple temporal averaging inadequate for modeling perceived quality.
  • Content-aware features: Content-aware features are extracted from CNN models pretrained for image classification because they encode video content and remain distortion-sensitive.The implementation uses ResNet-50 features from the res5c layer, producing 4096-dimensional frame features.
  • Temporal-memory effects: Temporal-memory effects are modeled with both a global GRU for long-term dependencies and a local temporal-pooling module over a window of 2τ + 1 frames.The two modules jointly capture long-term dependencies and local temporal hysteresis behavior.
  • Long-term dependencies modeling: A fully connected layer reduces the high-dimensional frame features before a GRU integrates them and predicts frame-wise quality scores.The GRU hidden states serve as integrated features, followed by a fully connected layer for frame-quality prediction.
  • Subjectively-inspired temporal pooling: The pooling model uses the minimum quality over previous frames as memory and weighted future-frame quality, emphasizing worse frames, as the current element.The current element uses differentiable softmin weights, allowing the pooling operation to be embedded in the neural network.
  • Subjectively-inspired temporal pooling: The approximate frame quality linearly combines memory and current elements, and temporal global average pooling produces the overall video quality.The parameter γ balances the contributions of the two elements.

4 EXPERIMENTS

Experiments evaluate the proposed method on three in-the-wild video quality databases against five no-reference baselines, then examine components, feature extractors, temporal pooling, motion information, and efficiency. The method achieves the best overall and database-specific performance, while ablations support the importance of content-aware features and temporal-memory modeling.

  • Performance comparison: The proposed method achieves the best overall performance for SROCC, KROCC, PLCC, and RMSE, with a large gain over VBLIINDS.SROCC and KROCC assess prediction monotonicity, whereas PLCC and RMSE assess prediction accuracy.
  • Performance comparison: 30.21% SROCC improvement over VBLIINDS occurs on LIVE-Qualcomm, alongside 8.63% on KoNViD-1k and 17.96% on CVD2014.LIVE-Qualcomm is the most challenging database for both compared methods and the proposed method, with low mean performance and large standard deviations.
  • Performance comparison: All paired t-test p-values are below 0.05, indicating significantly better SROCC for the proposed method than each of the five compared methods on every database.The tests use SROCC values from 10 runs at a 5% significance level.
  • Ablation study: 14.57%, 30.00%, and 26.87% SROCC decreases occur on KoNViD-1k, CVD2014, and LIVE-Qualcomm after replacing CNN content-aware features with BRISQUE features.The reported p-values are 1.10E-05, 1.76E-08, and 2.47E-06, respectively.
  • Ablation study: 7.70%, 4.14%, and 12.01% SROCC gains from temporal modeling are reported on KoNViD-1k, CVD2014, and LIVE-Qualcomm, respectively.Removing long-term dependency modeling or subjectively-inspired temporal pooling decreases KoNViD-1k SROCC by 2.12% and 2.68%, respectively; ResNet-50 performs similarly to VGG16 with fewer parameters than AlexNet and VGG16.

5 CONCLUSION AND FUTURE WORK

The proposed NR-VQA method incorporates content-dependency and temporal-memory effects, outperforming state-of-the-art methods across three in-the-wild video databases while leaving room for stronger human-perception correlation.

  • 30.21%, 8.63%, and 17.96% SROCC improvements over the second-best method VBLIINDS were achieved on LIVE-Qualcomm, KoNViD-1k, and CVD2014, respectively.
  • Content-aware perceptual features and temporal-memory modeling are important for in-the-wild video quality assessment.
  • The best method's correlation values remain below 0.76 on KoNViD-1k and LIVE-Qualcomm.
  • Future work will consider spatio-temporal attention models to provide information about when and where video content is important for VQA.
Loading 1908.00375v3…