PaperlayerView exact revision on arXiv

Source-linked AI summary

Perceptual Video Quality Assessment: A Survey

Xiongkuo Min, Huiyu Duan, Wei Sun, Yucheng Zhu, Guangtao Zhai

arXiv:2402.03413v1cs.MMcs.CVeess.IV

TL;DR

Video quality assessment matters because acquisition, compression, transmission, and display degradations can impair user experience amid expanding video traffic. This survey reviews subjective methodologies and databases, objective metrics for general and application-specific settings, and comparative performance, providing a systematic overview of classical and recent VQA research.

  • Problem

    Video quality can degrade across acquisition, compression, transmission, and display, creating a need for accurate assessment as video traffic expands and poor quality can reduce user engagement.

  • Method

    The survey reviews subjective VQA methodologies and databases, summarizes full-reference, reduced-reference, and no-reference metrics, covers emerging applications, and compares objective model performance.

  • Results

    The survey reports that deep learning-based VQA methods outperform traditional models in full-reference and no-reference comparisons, while VQA models outperform IQA models for no-reference VQA.

  • Takeaways & Limitations

    The survey offers researchers a systematic overview of classical and recent VQA progress across general-purpose and emerging application areas.

  • Takeaways & Limitations

    High-quality VQA labels for training robust no-reference models are time-consuming and expensive to obtain because they typically require subjective experiments.

Abstract

from arXiv · show

Perceptual video quality assessment plays a vital role in the field of video processing due to the existence of quality degradations introduced in various stages of video signal acquisition, compression, transmission and display. With the advancement of internet communication and cloud service technology, video content and traffic are growing exponentially, which further emphasizes the requirement for accurate and rapid assessment of video quality. Therefore, numerous subjective and objective video quality assessment studies have been conducted over the past two decades for both generic videos and specific videos such as streaming, user-generated content (UGC), 3D, virtual and augmented reality (VR and AR), high frame rate (HFR), audio-visual, etc. This survey provides an up-to-date and comprehensive review of these video quality assessment studies. Specifically, we first review the subjective video quality assessment methodologies and databases, which are necessary for validating the performance of video quality metrics. Second, the objective video quality assessment algorithms for general purposes are surveyed and concluded according to the methodologies utilized in the quality measures. Third, we overview the objective video quality assessment measures for specific applications and emerging topics. Finally, the performances of the state-of-the-art video quality assessment measures are compared and analyzed. This survey provides a systematic overview of both classical works and recent progresses in the realm of video quality assessment, which can help other researchers quickly access the field and conduct relevant research.

1 Introduction

Video quality assessment is needed because acquisition, compression, transmission, and display can degrade video and harm users’ quality of experience. This survey updates earlier reviews by covering deep-learning methods and application-specific VQA topics.

  • Video degradations can reduce quality of experience and engagement, making VQA important for delivered video content.
  • Subjective VQA is reliable but time-consuming and expensive, so it mainly evaluates objective metrics rather than serving communication systems directly.
  • Different video types have distinct perceptual characteristics, complicating objective VQA design across generic and specialized applications.
  • Earlier VQA surveys largely emphasized classical methods or specific topics, while deep neural networks and emerging applications were rarely comprehensively addressed.
  • The survey reviews subjective methodologies and databases, objective FR/RR/NR metrics, emerging applications, evaluation performance, and future trends.

2 Subjective Video Quality Assessment

Subjective quality assessment is treated as the perceptually reliable basis for evaluating video quality and validating objective VQA measures. The survey reviews standardized methodologies and general-purpose subjective databases.

  • Subjective assessment is considered the most reliable method because human eyes are the ultimate receivers of video content.
  • The survey reviews ITU-R BT.500 viewing environments, subject recruitment, grading, result processing, and 20 general-purpose subjective VQA databases.

2.1 Subjective VQA Methodology

Subjective VQA aggregates human ratings under controlled or crowdsourced procedures. The methodology spans score definitions, stimulus presentation, participant recruitment, rating scales, and scalable alternatives.

  • Subjective studies aggregate many participants’ ratings into mean opinion scores or difference mean opinion scores.
  • ACR suits overall unbiased judgments, whereas DCR and CCR are better suited to detecting smaller, subtle quality differences.
  • ITU-recommended experiments establish and calibrate the viewing environment, prepare stimuli, recruit participants, conduct ratings, and process results.
  • Crowdsourcing provides faster, cheaper, more scalable, and more diverse participation than recruiting numerous laboratory subjects.

2.2 Subjective VQA Databases for General Purpose

General-purpose VQA databases include laboratory datasets with synthetic distortions and larger authentic-distortion UGC datasets. They vary in content, resolutions, frame rates, formats, scores, and experimental environments.

  • Table 1 summarizes database types, content and sequence counts, video properties, distortions, subjects, ratings, scores, and assessment environments.
  • General VQA Databases with Synthetic Distortions: Synthetic databases commonly introduce compression, transmission, or scaling distortions under laboratory conditions, using MOS, DMOS, raw scores, or JND data.
  • General VQA Databases with Synthetic Distortions: LIVE-VQA contains 10 pristine and 160 distorted videos at 768×432 resolution, with 25 or 50 fps and DMOS plus σ scores.
  • General VQA Databases with Authentic Distortions: Recent authentic-distortion databases target in-capture or in-the-wild quality variation, often emphasizing diverse UGC content and crowdsourced-scale collections.

2.3 Subjective VQA Databases for Specific Applications

The survey reviews public subjective VQA databases for specialized applications, emphasizing streaming, 3D, and immersive VR video collections with varied distortions, resolutions, durations, and subject studies.

  • Specific-application VQA databases extend beyond generic videos to streaming, 3D, and immersive VR content.
  • Streaming VQA Databases: Streaming databases represent compression, stalling, frame drops, rate adaptation, packet loss, buffering, rebuffering, and ABR-related degradations.
  • Streaming VQA Databases: Waterloo streaming collections include diverse combinations of encoders, network traces, ABR algorithms, devices, resolutions, frame rates, and bandwidth conditions.
  • 3D VQA Databases: 3D databases cover compression, depth variation, geometric distortion, packet loss, frame-rate changes, frame freeze, and camera-distance stimuli.
  • VR VQA Databases: Immersive VR databases use omnidirectional videos viewed through head-mounted displays and include distortions from compression, resolution, and frame-rate changes.

3 Objective Video Quality Assessment: General-purpose Models

General-purpose objective VQA models are organized by the amount of reference-video information they require. The survey distinguishes full-reference, reduced-reference, and no-reference approaches, with FR methods commonly measuring fidelity against pristine video.

  • Objective VQA models are categorized as full-reference, reduced-reference, or no-reference according to reference-video availability.
  • Full-reference models compare distorted video with its original or pristine counterpart and are used across broadcasting, streaming, compression, enhancement, and production quality control.
  • A common FR strategy applies image-quality measures to sampled video frames and aggregates frame-level scores into a video-level score.

3.1.1 Knowledge-driven FR VQA

Knowledge-driven VQA methods use manually designed perceptual or statistical features, while FR approaches increasingly combine spatial, temporal, motion, HVS, and learned feature representations.

  • SSIM-based FR VQA: SSIM-based FR VQA extends frame-level luminance, contrast, and structure comparisons from image quality assessment into video quality assessment.
  • Low-level feature-based FR VQA: MOVIE uses Gabor-filter decomposition and optical-flow motion estimation to derive spatial, temporal, and motion features for video integrity evaluation.
  • HVS-based FR VQA: Other knowledge-driven FR models address flicker, motion complexity, visual attention, saliency, and HVS-related perceptual distortions.
  • Features fusion based FR VQA: Feature-fusion FR models combine heterogeneous descriptors and use learning-based regressors to map them to video quality scores.
  • Reduced-reference models use partial reference information to reduce transmission bandwidth, whereas no-reference models operate when reference videos are unavailable.
  • Knowledge-driven NR VQA: Knowledge-driven NR methods use handcrafted spatial, temporal, natural-scene, and motion features, including the feature designs used by Video BLIINDS and related metrics.
  • Data-driven NR VQA: Data-driven NR VQA methods require large-scale datasets, motivating weakly supervised, self-supervised, and unsupervised approaches for learning from labeled or unlabeled video data.

4 Objective Video Quality Assessment: Specific-purpose Models

Specific-purpose VQA research addresses compressed and streaming video as well as emerging application domains. The survey highlights reference-based and no-reference methods alongside QoS- and visual-quality-driven QoE models.

  • Emerging VQA topics include compressed, streaming, stereoscopic, VR, frame-rate, audio-visual, HDR/WCG, screen, and game video.
  • Compressed VQA: Compressed-video assessment uses FR or RR comparisons with uncompressed video and NR methods that infer quality from compressed bitstream decoding parameters.
  • Streaming VQA: Streaming QoE modeling is challenged by variable environments and complex human responses to delivery conditions.
  • QoS and QoE-driven User QoE Assessment: QoS-driven models analyze relationships among client, coding, CDN, and streaming factors to assess or optimize user QoE.
  • Data-driven QoE Assessment: Data-driven QoE models use machine-learning methods such as random forests and neural networks for no-reference monitoring of HTTP/TCP video streaming.

4.3 Stereoscopic VQA

Stereoscopic VQA extends 2D quality assessment while increasingly modeling binocular perception and motion–disparity relationships. The surveyed methods span full-, reduced-, and no-reference settings.

  • 3D video quality assessment remains important because stereoscopic content, capture, and display devices still limit optimal visual experiences.
  • Traditional approaches apply IQA or VQA algorithms separately to stereoscopic views, including disparity views, and aggregate frame-level estimates.
  • Full-reference methods incorporate binocular suppression, recurrent excitation, and temporal pooling to model stereoscopic perception.
  • Reduced-reference metrics assess color-plus-depth transmission using edge information from depth maps and nearby color-image regions.
  • No-reference models use disparity entropy, energy-weighted content features, or motion–disparity dependencies to predict 3D video quality.

4.4 VR VQA

VR video quality assessment must account for spherical viewing, projection distortion, viewport selection, and human visual behavior. Recent deep-learning methods address these factors in both reference-based and no-reference designs.

  • Omnidirectional video metrics address spherical viewing and projection distortion through distortion weighting, resampling, or low-distortion viewport extraction.
  • Traditional VR quality methods weight projection-plane pixel errors by equivalent spherical area to improve evaluation reliability.
  • Human viewing preferences and temporal perception are incorporated through front-region weighting, viewport sampling, spatio-temporal metrics, and regression.
  • Deep-learning FR methods combine viewport proposal, saliency prediction, and viewport-quality networks for 360° video assessment.
  • NR VR methods fuse local spatiotemporal features or combine spherical convolutions with non-local neural operations to predict quality.

4.5 Framerate & Frame Interpolation VQA

Frame rate and interpolation introduce temporal quality factors that require specialized assessment. The surveyed work addresses motion smoothness and interpolation-specific artifacts using temporal statistics and human-visual considerations.

  • Frame-rate changes affect choppiness, motion smoothness, clarity, and realism, motivating dedicated framerate VQA methods.
  • Framerate VQA: Temporal video quality can be represented using statistics from bandpass-filtered videos.
  • Frame Interpolation VQA: Frame interpolation requires specialized metrics because its artifacts can create inconsistencies between common quality metrics and human perception.

4.6 Audio-Visual VQA

Audio-visual quality assessment addresses distortions affecting jointly perceived audio and video quality, while emerging video formats such as HDR require specialized assessment because they differ from SDR.

  • Audio-Visual VQA: Compression of audio and video signals can distort end-user quality of experience, making joint AVQA an important research area.
  • Audio-Visual VQA: Previous work often focused on single modalities, while newer AVQA studies jointly model audio and visual quality.
  • Audio-Visual VQA: A no-reference audiovisual model extracts audio features with separable convolution, visual features with quality-aware ResNet-50, and temporal information with Bi-LSTM.
  • HDR VQA: HDR video quality assessment faces new challenges because HDR characteristics differ from SDR and require specialized models.
  • HDR VQA: HDR research covers subjective datasets, streaming factors, compression effects, luminance conversion, frequency and orientation analysis, and tone mapping.

4.8 Screen and Game VQA

Screen content videos differ from natural scene videos, while gaming videos have received limited automatic quality-prediction research. Existing approaches address these domains with specialized feature extraction and fusion strategies.

  • Screen Content VQA: Screen content videos exhibit distinct characteristics from natural scene videos and have become a research focus.
  • Screen Content VQA: Li et al. proposed a full-reference screen content VQA model using frame differences to measure relative standard deviation similarity between reference and distorted content.
  • Game VQA: Gaming videos have grown substantially, but automatic quality prediction for this content has received limited research.
  • Game VQA: The proposed gaming-video method extracts low-level natural scene statistics and high-level CNN features, then trains separate SVR models and fuses their predictions.

5 Objective Video Quality Assessment Model Evaluation

Objective VQA models are evaluated by fitting predicted scores to subjective ratings and measuring accuracy, monotonicity, and consistency with traditional statistical metrics. Comparisons indicate that temporal features, deep learning, and video-specific models improve performance across evaluated settings.

  • Evaluation Criteria: Objective VQA evaluation fits objective scores with a five-parameter logistic function before comparing them with ground-truth subjective ratings.The fitted function uses s and q(s) for objective and best-fitting quality, with β1 through β5 fitted during evaluation.
  • Evaluation Criteria: SRCC, KRCC, and PLCC measure correlation-related prediction monotonicity, whereas RMSE and MAE measure prediction accuracy.
  • Evaluation Criteria: Higher SRCC, KRCC, and PLCC values closer to 1, and lower RMSE and MAE values closer to 0, indicate better performance.
  • Performance Comparison: Because not all surveyed algorithms are publicly available, the comparison uses performance values reported in the original papers.
  • Performance Comparison: On LIVE VQA, general-purpose FR-IQA measures perform worse than FR-VQA metrics, indicating the usefulness of hand-crafted temporal features.
  • Performance Comparison: On LIVE VQA, DeepVQA achieves better performance than traditional VQA models, demonstrating the effectiveness of DNNs for VQA.
  • Performance Comparison: Across KoNViD-1k, LIVE-VQC, and YouTube-UGC, traditional NR-IQA measures perform worse than deep NR-IQA models, while VQA models outperform IQA models.

6 Future Research Directions

Future VQA research should better model human perception, adapt to emerging media and AI-generated videos, support immersive multimodal assessment, and reduce computational cost for deployment.

  • 6.1 Human Perception Mechanism of Video Quality Assessment: Perception-based VQA models can improve the interpretability and robustness of current systems while supporting reliable results across new media technologies.The paper emphasizes integrating knowledge of human visual perception as capture, compression, transmission, and display techniques evolve.
  • 6.2 Large Multi-modality Models for Video Quality Assessment: LMM-based quality assessment performs strongly for images but still trails state-of-the-art VQA because current methods overlook temporal video distortions.The paper identifies video-specific temporal characteristics as a primary reason for this performance gap.
  • 6.3 Quality Assessment of Emerging Video Media: Emerging media such as VR/AR/MR, HFR, HDR, and gaming require application-specific VQA systems and expanded multimodal datasets.For XR, future assessments should include visual, auditory, olfactory, gustatory, and tactile modalities while accounting for immersive and interactive content.
  • 6.4 Quality Assessment of AIGC Videos: Quality assessment for AI-generated videos is a growing research direction because their distinctive distortions require further study.The paper links this work to controlling and improving the quality of AI-generated videos.
  • 6.5 Quality Assessment of Volumetric Videos: Volumetric video remains understudied compared with static 3D content despite its potential to provide immersive and realistic experiences.The paper describes volumetric video as dynamic 3D content representing the complete volume of a scene or object.
  • 6.6 Green Learning for Video Quality Assessment: Green learning targets lightweight, low-complexity, logically transparent VQA models suitable for edge devices and real-time processing.Existing DNN-based models are described as difficult to deploy in these settings because of their large size and high computational complexity.

7 Summary

The survey systematically reviews subjective methodologies, objective metrics, emerging applications, and model performance in perceptual video quality assessment. It is intended to help researchers access established progress and identify relevant solutions and trends.

  • 7 Summary: The survey reviews subjective VQA methodologies and databases before summarizing FR, RR, and NR objective metrics.It covers both general-purpose assessment and emerging application areas.
  • 7 Summary: It surveys objective VQA for compressed, streaming, stereoscopic, VR/AR, HFR, audio-visual, HDR, screen, and game videos.These topics extend the review beyond traditional general-purpose metrics.
  • 7 Summary: The survey evaluates and compares the performance of many objective video quality assessment models.This comparison follows the review of subjective and objective assessment research.
  • 7 Summary: The resulting overview covers classical and recent VQA progress to help researchers access the field and find solutions and trends.The supplied summary describes this as the survey's practical research utility.
Loading 2402.03413v1…