Source-linked AI summary
Patch-VQ: 'Patching Up' the Video Quality Problem
Zhenqiang Ying, Maniratnam Mandal, Deepti Ghadiyaram, Alan Bovik
TL;DR
Real-world UGC makes no-reference video quality assessment difficult because distortions are diverse, composite, and often temporal, while existing datasets and models are limited. The paper introduces a large annotated UGC dataset with localized v-patches and uses it to build PVQ and PVQ Mapper. PVQ achieves state-of-the-art performance across three UGC datasets, while PVQ Mapper localizes and visualizes distortions in space and time.
Problem
NR video quality prediction lacks pristine references and remains limited on real-world UGC because distortions are diverse, composite, perceptually nonuniform, and often temporal.
Method
The paper constructs a large UGC database with full videos, three kinds of v-patches, and subjective ratings, then learns local-to-global quality with PVQ and space-time maps with PVQ Mapper.
Results
PVQ achieves state-of-the-art performance on three UGC datasets, including a 2.8% SRCC improvement over the strongest baseline on LIVE-VQC.
Takeaways & Limitations
Patch-level annotations enable global-to-local and local-to-global quality inferences, supporting accurate and generalizable prediction and space-time distortion mapping.
Takeaways & Limitations
Subjective ratings limited the dataset to one 40%-scale for each v-patch type, and some content cases remained misaligned with human judgments.
Abstract
from arXiv · showhide
No-reference (NR) perceptual video quality assessment (VQA) is a complex, unsolved, and important problem to social and streaming media applications. Efficient and accurate video quality predictors are needed to monitor and guide the processing of billions of shared, often imperfect, user-generated content (UGC). Unfortunately, current NR models are limited in their prediction capabilities on real-world, "in-the-wild" UGC video data. To advance progress on this problem, we created the largest (by far) subjective video quality dataset, containing 39, 000 realworld distorted videos and 117, 000 space-time localized video patches ('v-patches'), and 5.5M human perceptual quality annotations. Using this, we created two unique NR-VQA models: (a) a local-to-global region-based NR VQA architecture (called PVQ) that learns to predict global video quality and achieves state-of-the-art performance on 3 UGC datasets, and (b) a first-of-a-kind space-time video quality mapping engine (called PVQ Mapper) that helps localize and visualize perceptual distortions in space and time. We will make the new database and prediction models available immediately following the review process.
1. Introduction
NR video quality assessment is difficult because real-world UGC contains complex spatial and temporal distortions, while existing datasets and models remain limited. The paper addresses these gaps with a large annotated dataset and models linking local and global perceptual quality.
- Motivation: Billions of daily Internet-video views make accurate UGC quality measurement valuable for capture, processing, sharing, and analysis.The motivation spans social media platforms and other video-processing contexts.
- Challenges: NR quality prediction remains largely unsolved because it lacks pristine references and must handle diverse, composite distortions whose technical severity may not match perceived quality.Equal amounts of blur, blocking, or noise can affect different videos very differently because of perceptual masking.
- Challenges: Existing resources are too small and unrepresentative for training modern deep models, while large subjective studies are substantially harder and more time-consuming than standard classification data collection.Available authentically distorted UGC databases are described as too small for data-hungry neural networks.
- Challenges: Most existing NR algorithms are frame-based or use sampled frame differences, limiting their ability to capture transient and diverse temporal impairments in UGC.Examples include frame drops, focus changes, and transmission glitches.
- Contributions: The paper contributes 39,000 real-world videos, 117,000 space-time v-patches, and 5.5M perceptual judgments from almost 6,300 subjects.The dataset was designed to match feature distributions of social-media UGC and is substantially larger than prior UGC resources and studies.
- Contributions: PVQ learns local-to-global quality relationships using parallel 2D and 3D features, while PVQ Mapper produces space-time quality maps to localize and visualize distortions.The proposed models predict global and local quality and support temporal quality series and space-time quality maps.
2. Related Work
Prior video-quality resources and models often perform well on controlled legacy datasets but struggle with the diverse, mixed distortions and limited scale of in-the-wild UGC data.
- Video Quality Datasets: Legacy datasets typically contain only 10–15 source videos with a few manually applied synthetic impairments, limiting content diversity and distortion complexity.Examples include Gaussian blur, compression, and transmission artifacts.
- Video Quality Datasets: In-the-wild databases contain complex mixtures of diverse real distortions, but the available resources are much smaller than needed for modern deep learning.The dataset comparison distinguishes these databases from legacy collections with singular synthetic distortions.
- NR-VQA Models: Existing NR models mostly perform well on legacy datasets and struggle on in-the-wild UGC databases.The cited comparison includes frame-feature and recurrent approaches, while MLSP-VQA results were reported as unreproducible by the authors.
3. Large-Scale Dataset and Human Study
The LSVQ Database contains 39,075 diverse UGC videos and 117,225 space-time patches, supported by a large AMT study with extensive participant screening and consistency checks.
- Dataset construction: 39,075 videos, 117,225 v-patches, and about 5.5M quality scores from around 6,300 subjects form the LSVQ Database.The dataset is reported as 26 times larger than the YouTube-UGC dataset.
- Dataset construction: 400,000 videos from the Internet Archive and YFCC100M were sampled and feature-matched to produce roughly 39,000 representative UGC videos.The collection preserved diverse resolutions, aspect ratios, contents, and distortions.
- Dataset construction: Three v-patch types capture spatial, temporal, and joint space-time crops, each remaining a video with the source aspect ratio.Each patch is contained within its source, and same-source sv-patch/tv-patch overlap with the stv-patch was limited to 25%.
- Human study: AMT sessions collected about 35 ratings per video and v-patch from 6,284 participants across separate video and patch tasks.Participants completed instructions, quizzes, training, and testing; testing included 90 videos rated on a 0–100 quality scale.
- Human study: 1,046 subjects were rejected through browser, playback, training, and data-quality checks, followed by additional score cleaning and standard screening.The cleaned ratings achieved average SRCC values of 0.86 for full videos and 0.71, 0.71, and 0.67 for sv-, tv-, and stv-patches.
4. Modeling a Blind Video Quality Predictor
The paper introduces PVQ, a blind video-quality predictor that models local-to-global space-time quality from unprocessed videos. Its pipeline extracts spatial and temporal features, pools regional information, and regresses quality over time.
- Model overview: PVQ and PVQ-Mapper exploit the LSVQ Database to predict video quality and map perceptual distortions in space and time.PVQ targets prediction, while PVQ-Mapper produces space-time quality maps.
- Model overview: PVQ processes original videos without cropping or subsampling, accommodating diverse aspect ratios, resolutions, and durations.Avoiding preprocessing prevents additional spatial or temporal artifacts, while direct end-to-end training would otherwise be impractical.
- Feature extraction: 2D spatial features capture frame content and perceptual quality, while 3D features capture temporal distortions such as flicker, stutter, and focus changes.The streams use PaQ-2-PiQ and 3D ResNet-18 backbones, respectively.
- Spatio-temporal pooling: Spatial RoIPool uses v-patch coordinates to produce fixed-size regional feature maps for full videos and three patch types.The pooled representations support modeling local-to-global space-time quality relationships.
- Temporal regression: InceptionTime learns temporal changes in space-time quality features before global average pooling and fully connected regression.Temporal modeling is used because changes over time are crucial for predicting global video quality.
5. Experiments
Experiments evaluate PVQ and PVQ Mapper on LSVQ and smaller in-the-wild datasets, testing global and local quality prediction, ablations, mapping quality, and cross-database transfer.
- Experimental setup: The dataset uses training and two test sets, including a high-resolution Test-1080p split with 3.5K videos and 10.5K v-patches.The remaining data provide about 28.1K training videos and 7.4K test videos.
- Global video quality: PVQ trained with v-patches outperforms the strongest baseline by 3.6% on SRCC for Test-1080p videos.Including v-patches improves performance on both test sets.
- Ablation studies: PVQ models deliver the best performance on all three v-patch types, while PVQ3D outperforms PVQ2D on both test sets.Among individual patch types, stv-patches produce the highest performance despite their relatively small volume.
- Failure cases: Failure cases include bokeh blur judged high quality by subjects but predicted low quality, and a game video overrated by PVQ with predicted MOS 54.7 versus human MOS 21.These cases illustrate challenges in aligning predictions with human perception across diverse content.
- PVQ Mapper: PVQ Mapper generates temporal quality series and space-time maps that distinguish blurred, underexposed, stationary, and fast-moving regions.Changing clip length controls the density of quality predictions.
- PVQ Mapper: Using v-patches makes local distortions distinctly evident in PVQ Mapper outputs that are not captured without v-patch training.The examples include focus blur, motion blur, and underexposure.
- Cross-database comparisons: PVQ improves SRCC over the strongest baseline by 2.8% on LIVE-VQC when models are separately trained and tested on each database.PVQ competes well on KoNViD-1k under the same protocol.
- Cross-database comparisons: When trained on LSVQ without fine-tuning, PVQ improves SRCC by 0.7% on KoNViD-1k and 3.6% on LIVE-VQC over the strongest baseline.The authors associate this database independence with LSVQ’s representativeness and PVQ’s general efficacy.
6. Concluding Remarks
The paper introduces LSVQ, PVQ, and PVQ Mapper to advance quality prediction for user-generated videos through large-scale data, patch-level annotations, and deep learning.
- Concluding Remarks: LSVQ’s patch-level annotations support global-to-local and local-to-global quality inferences for UGC video quality assessment.The database is described as substantially larger and more diverse than previous resources.
- Concluding Remarks: PVQ predicts global video quality, while PVQ Mapper infers local space-time quality and generates spatio-temporal quality maps.The authors state that these resources can support quality-based monitoring, ingestion, and control of streamed videos.
A. Cropping Patches
The dataset extracts video patches from space-time volumes, using one scale for each patch type because subjective ratings are time-consuming.
- Cropping Patches: Only one scale per v-patch type was collected because subjects require roughly 15–20 seconds to rate each video-quality content.The selected scale uses 40% of the original dimensions.
- Cropping Patches: Each video patch is a video crop extracted from a random space-time volume.The examples show triplets of v-patches from exemplar videos.
B.1 Inter-subject consistency plots:
Inter-subject agreement is generally high, but consistency is lowest for stv-patches.
- Inter-subject consistency plots: Scatter plots show narrow spread and high agreement between subject MOS ratings, with the widest spread for stv-patches.The lower consistency is attributed to stv-patches accounting for only 6.4% of video pixel volume.
B.2 Consistency among subject demographics:
Subjective ratings were broadly consistent across devices, resolutions, viewing distances, age groups, and genders, while playback delays had negligible impact on mean scores. Distribution-aware outlier rejection further improved consistency.
- Device parameters: 0.7 SRCC between laptop and desktop MOS, versus 0.5 between phone and other-device ratings, indicates viewing device affects perceived quality.Phone data comprised 3.7% of observations, reinforcing the relevance of small-screen viewing despite limited phone samples.
- Subject demographics: 0.76 correlations across major resolutions and viewing-distance groups, 0.79 across age groups, and 0.8 across genders show high cross-group rating consistency.The paper characterizes these correlations as high but still suggestive of further study.
- Playback delays: More than 96% of videos had playback delays below 1 second, and 86% played without delays.Comparisons with golden scores found negligible delay effects on mean scores and no consistency change after removing delayed-video ratings.
- Outlier rejection: Distribution-adaptive outlier rejection improved consistency by applying modified Z-scores to approximately Gaussian scores and Tukey IQR detection otherwise.Kurtosis was used to determine score-distribution normality before selecting the rejection method.