PaperlayerView exact revision on arXiv

Source-linked AI summary

Unified Quality Assessment of In-the-Wild Videos with Mixed Datasets Training

Dingquan Li, Tingting Jiang, Ming Jiang

arXiv:2011.04263v2cs.CVcs.MMeess.IV

TL;DR

In-the-wild VQA must handle absent pristine references, complex distortions, diverse contents, and cross-dataset differences. The paper trains a unified mixed-dataset model with three quality-prediction stages, and reports superior monotonicity and accuracy across four datasets. Its scope includes retraining when new datasets are added and remaining failures on rapidly changing scenes.

  • Problem

    In-the-wild VQA is challenging because pristine references are unavailable, distortions are complex, contents are diverse, and existing methods perform poorly across datasets.

  • Method

    The unified model uses a relative quality assessor, nonlinear mapping, and dataset-specific perceptual scale alignment, trained with mixed datasets and human-perception-inspired modules.

  • Results

    The method achieves superior prediction monotonicity and accuracy across KoNViD-1k, CVD2014, LIVE-Qualcomm, and LIVE-VQC.

  • Takeaways & Limitations

    Mixed datasets training enables a single unified VQA model for multiple datasets while using their existing relevant data.

  • Takeaways & Limitations

    The strategy requires retraining the unified model whenever a new dataset is added, increasing training burden.

Abstract

from arXiv · show

Video quality assessment (VQA) is an important problem in computer vision. The videos in computer vision applications are usually captured in the wild. We focus on automatically assessing the quality of in-the-wild videos, which is a challenging problem due to the absence of reference videos, the complexity of distortions, and the diversity of video contents. Moreover, the video contents and distortions among existing datasets are quite different, which leads to poor performance of data-driven methods in the cross-dataset evaluation setting. To improve the performance of quality assessment models, we borrow intuitions from human perception, specifically, content dependency and temporal-memory effects of human visual system. To face the cross-dataset evaluation challenge, we explore a mixed datasets training strategy for training a single VQA model with multiple datasets. The proposed unified framework explicitly includes three stages: relative quality assessor, nonlinear mapping, and dataset-specific perceptual scale alignment, to jointly predict relative quality, perceptual quality, and subjective quality. Experiments are conducted on four publicly available datasets for VQA in the wild, i.e., LIVE-VQC, LIVE-Qualcomm, KoNViD-1k, and CVD2014. The experimental results verify the effectiveness of the mixed datasets training strategy and prove the superior performance of the unified model in comparison with the state-of-the-art models. For reproducible research, we make the PyTorch implementation of our method available at https://github.com/lidq92/MDTVSFA.

1 Introduction

In-the-wild VQA is difficult because videos lack pristine references, contain diverse content and complex distortions, and existing datasets differ in quality-score ranges. The paper addresses cross-dataset evaluation with a unified model that separates relative, perceptual, and subjective quality.

  • Motivation: In-the-wild VQA lacks pristine references and must handle diverse contents and complex, temporally heterogeneous real-world distortions.These conditions make methods validated on synthetically distorted videos perform poorly on in-the-wild videos.
  • Motivation: Cross-dataset performance is poor because existing datasets differ in video contents and distortions.The paper identifies mixed datasets training as a way to learn characteristics shared across datasets.
  • Motivation: Mixed datasets training can provide one model for multiple datasets and use more existing annotations, but inconsistent subjective-score ranges require alignment.The paper contrasts learned alignment with additional subjective studies, which it describes as time-consuming and impracticable as datasets grow.
  • Motivation: Linear rescaling can assign identical labels to videos whose inherent qualities differ, disturbing training.The paper illustrates this mismatch with the worst-quality videos from CVD2014 and LIVE-VQC.
  • Proposed solution: The framework decomposes VQA into predicting relative quality, perceptual quality, and subjective quality in sequence.Relative quality supports ranking, perceptual quality represents an ideal human-perception target, and subjective quality is obtained from limited dataset studies.
  • Results: Experiments on four public in-the-wild VQA datasets verify superior prediction monotonicity and accuracy across datasets.The four datasets are KoNViD-1k, CVD2014, LIVE-Qualcomm, and LIVE-VQC.
  • Novelty: Compared with the previous work, this paper adds mixed datasets optimization and explicitly introduces three stages for the three quality concepts.The earlier model provides the basis, but does not consider mixed datasets training.

2 Related Work

Related work covers classical and learned VQA methods, in-the-wild datasets, and mixed-dataset training. Existing in-the-wild methods perform well on individual benchmarks but generally do not support training across multiple datasets.

  • VQA methods: Classical VQA methods use cues including structure, motion, energy, saliency, gradients, and natural video statistics.Other methods combine primary features or learn spatio-temporal representations.
  • In-the-wild VQA: Four public datasets support in-the-wild VQA: CVD2014, KoNViD-1k, LIVE-Qualcomm, and LIVE-VQC.Because pristine references are unavailable, the related work focuses on no-reference VQA methods.
  • In-the-wild VQA: Recent methods use low- and high-complexity frame features, 3D-CNNs with LSTMs, or HVS-inspired content-aware and distortion-sensitive features.The previous HVS-inspired model also uses recurrent modeling and temporal pooling for temporal-memory effects.
  • Cross-dataset evaluation: Existing in-the-wild methods can perform well on individual benchmarks but perform poorly in cross-dataset evaluation and do not enable mixed-dataset training.The related work identifies this limitation across the cited methods.
  • Mixed datasets training: Mixed datasets training offers a unified model and fuller use of relevant datasets for model learning.The paper places its linearity-induced loss and dataset-specific perceptual scale alignment within this training direction.

3.1 Overview

The unified framework models three related quality targets in sequence, combining human-perception-inspired assessment with nonlinear mapping and dataset-specific scale alignment. Mixed-dataset optimization combines three losses into a single training objective for one model across datasets.

  • Relative quality assessor: The relative quality assessor uses monotonicity-induced loss and models content dependency and temporal-memory effects.Its design includes content-aware feature extraction and temporal-memory modeling at both feature and quality-score levels.
  • Nonlinear mapping: A nonlinear mapping module predicts perceptual quality from relative quality using a Pearson-correlation-based linearity-induced loss.This stage explicitly accounts for the nonlinearity of human perception.
  • Perceptual scale alignment: Dataset-specific perceptual scale alignment maps predicted perceptual quality to each dataset’s subjective quality with an error-induced loss.The alignment addresses differences in subjective scales across datasets.
  • Mixed datasets training: The final training loss combines the three stage losses for each dataset and applies softmax weighting across training datasets.This strategy trains a single unified VQA model by mixing the datasets during training.

3.2 Relative Quality Assessor

The relative quality assessor models content dependency and temporal-memory effects to predict video quality rankings from frame-level features. It combines content-aware extraction, GRU-based temporal integration, and subjectively inspired temporal pooling.

  • Architecture: The assessor integrates content dependency and temporal-memory effects through content-aware feature extraction and temporal modeling.Content dependency motivates the feature extraction module, while temporal memory is modeled at feature and quality-score levels.
  • Content-aware feature extraction: A pre-trained CNN extracts frame features using global average and standard-deviation pooling, whose concatenation forms content-aware and distortion-sensitive representations.The resulting feature vector has length 2C.
  • Temporal-memory modeling: A fully connected layer reduces feature dimensionality before a GRU models long-term frame dependencies and produces integrated features for frame-quality prediction.The reduced features are sent sequentially to the GRU, followed by a fully connected layer that predicts frame quality.
  • Subjectively-inspired temporal pooling: The model combines memory and current quality elements, then applies temporal global average pooling and a sigmoid function to obtain the relative quality score.The memory element uses the minimum quality over previous frames, while the current element summarizes related future frames.
  • Subjectively-inspired temporal pooling: The subjectively inspired temporal-pooling model assigns larger weights to worse upcoming frames because subjects react sharply to quality drops but dully to improvements.The current quality element uses a differentiable softmin over related future frames.

3.3 Nonlinear Mapping

The nonlinear mapping module converts relative quality into perceptual quality while explicitly modeling the nonlinearity of human perception. It uses a four-parameter logistic mapping implemented as a network module without changing the score range.

  • Nonlinear mapping: The nonlinear mapping module follows the relative quality assessor to account explicitly for the nonlinearity of human perception on video quality.It may be implemented as either a complex neural network or a simple nonlinear function.
  • Logistic mapping: A 4-parameter logistic function maps relative quality score Qr to perceptual quality score Qp.The parameters β1 to β4 are fitted mapping parameters.
  • Network implementation: The logistic mapping is reformulated as Linear+Sigmoid+Linear so it can be implemented as a network module.The reformulation separates range control for Qp from normalization control for Qr.
  • Scale handling: The nonlinear mapping preserves the ranges of both Qr and Qp as [0, 1], leaving scale alignment to the following stage.The four parameters are initialized using priors about the relative and perceptual quality scores rather than randomly.

3.4 Dataset-Specific Perceptual Scale Alignment

The perceptual scale alignment maps perceptual quality to subjective quality while accommodating different subjective-score ranges across datasets. It uses dataset-specific scale and shift parameters learned jointly during mixed-dataset training.

  • Alignment model: Subjective quality is assumed to be linearly correlated with perceptual quality, so alignment is implemented as a fully connected layer.The layer represents the scale and shift between perceptual and subjective quality.
  • Dataset-specific alignment: Different datasets receive separate perceptual-scale alignments because their subjective quality scores have different ranges.The dataset-specific parameters are indexed by dataset and include one scale and one shift parameter.
  • Parameter learning: The alignment parameters can be estimated by least-square regression or jointly learned with the other network parameters using iterative SGD.The authors adopt joint learning because it provides supervision for end-to-end mixed-dataset training.

3.5 Mixed Datasets Training Strategy

Mixed-dataset training uses separate objectives for ranking, perceptual linearity, and subjective-score accuracy, then combines dataset losses into one unified-model objective. The strategy supports training across datasets with inconsistent subjective-score scales.

  • Training objectives: The first two stages avoid subjective-score range incompatibility by supervising relative quality and perceptual quality with separate losses.Relative quality targets monotonic ranking, while perceptual quality targets linearity with subjective quality.
  • Monotonicity-induced loss: The monotonicity-induced loss uses pairwise ranking errors to train relative quality scores consistent with subjective-quality ordering.The loss serves as a differentiable surrogate for rank-correlation objectives such as SROCC and KROCC.
  • Monotonicity-induced loss: List-wise learning reduces the number of forward operations from C2 Nd to Nd compared with pair-wise learning.Vectorization further improves implementation efficiency and saves training time.
  • Linearity-induced loss: The linearity-induced loss is based on PLCC because PLCC characterizes linearity and is differentiable for neural-network optimization.This loss supervises the nonlinear mapping stage.
  • Subjective-quality loss: After scale alignment, normalized L1 error is used for subjective-quality prediction, with inverse dataset scale balancing losses across datasets.The dataset score range is defined as Sd = max(Qd,i) − min(Qd,i).
  • Unified objective: The final objective is a softmax-weighted average of the losses from all datasets, enabling training of one unified model.Each dataset contributes a weighted loss to the overall mixed-dataset training objective.

3.6 Implementation Details

The implementation uses a content-aware feature extractor, dimensionality reduction, recurrent temporal modeling, temporal pooling, and nonlinear mapping.

  • ResNet-50 pretrained on ImageNet extracts content-aware features from its top convolutional layer, res5c.The resulting feature dimension is 4096.
  • The feature dimension is reduced from 4096 to 128 before a single-layer GRU with hidden size 32.
  • Temporal pooling uses τ=12 and γ=0.5, while the model adopts a 4-parameter nonlinear mapping.

4 Experiments

Experiments show that mixed-dataset training, dataset-specific scale alignment, and the proposed losses improve unified in-the-wild VQA across multiple datasets, while performance benefits from more similar training data and remains limited by unmodeled fast scene changes.

  • Loss combinations: The three-loss combination outperforms two-loss and single-loss combinations, with the linearity-induced loss contributing most to the gain.
  • Scale alignment: 0.7753 is the weighted median SROCC achieved by mixed-dataset training, improving over 0.7576 from linear rescaling.The paper attributes the further gain to dataset-specific perceptual scale alignment.
  • Mixed datasets: Adding a dataset generally improves overall performance, with CVD2014 adding a 0.1479 median-SROCC gain on its corresponding test set.Performance on existing datasets can decrease slightly after another dataset is added.
  • Cross-dataset effects: Mixed-dataset training improves performance when added data resemble the test distribution and can alleviate overfitting on the smallest dataset, LIVE-Qualcomm.
  • Training-data quantity: Performance increases with the training-data proportion, while the method still performs well when using one-half of the training data.
  • Performance comparison: The unified model achieves the best overall SROCC and PLCC among the compared methods across the reported datasets, with MDTVSFA slightly below TLVQM in CVD2014 PLCC.
  • Qualitative analysis: Scatter plots show MDTVSFA predictions most closely correlate with MOS across CVD2014, KoNViD-1k, and LIVE-Qualcomm.
  • Limitations: Failure cases indicate that frame-level features do not fully exploit motion and spatiotemporal information, especially during sudden fast scene changes.

5 Conclusion and Future Work

The paper proposes a unified NR-VQA framework using mixed datasets training and reports superior performance across four in-the-wild VQA datasets. Adding new datasets currently requires retraining the unified model, motivating future work on lifelong learning and broader video-production quality assessment.

  • The proposed unified NR-VQA framework combines mixed datasets training with HVS-inspired content-dependency and temporal-memory modeling for in-the-wild videos.It uses monotonicity-induced and linearity-induced losses together with dataset-specific perceptual scale alignment layers.
  • Experiments on four publicly available in-the-wild VQA datasets show superior performance and verify the effectiveness of the unified mixed-dataset model.
  • Adding a new dataset requires retraining the unified VQA model, while future directions include lifelong learning and using video metadata such as resolution.The authors also intend to extend the framework toward the broader video-production chain and applications such as video enhancement.
Loading 2011.04263v2…