PaperlayerView exact revision on arXiv

Source-linked AI summary

FAST-VQA: Efficient End-to-end Video Quality Assessment with Fragment Sampling

Haoning Wu, Chaofeng Chen, Jingwen Hou, Liang Liao, Annan Wang, Wenxiu Sun, Qiong Yan, Weisi Lin

arXiv:2207.02595v1cs.CVcs.MM

TL;DR

High-resolution deep VQA is costly, and naive resizing or cropping can corrupt quality information needed for representation learning. FAST-VQA combines Grid Mini-patch Sampling with a fragment-specific attention network, achieving higher accuracy and much lower computation on 1080P videos while supporting transfer to smaller datasets.

  • Problem

    Deep VQA methods incur high computational and memory costs on high-resolution videos, while resizing and cropping can corrupt quality-related information.

  • Method

    FAST-VQA uses Grid Mini-patch Sampling to create spatially spliced, temporally aligned fragments and FANet modules that process their patch relationships.

  • Results

    −99.5% FLOPs and +10% PLCC are achieved over PVQ on 1080P videos, while FAST-VQA also improves the fixed-feature Swin-T baseline by 8.10% PLCC on LSVQ1080p.

  • Takeaways & Limitations

    The learned video-quality-related representations improve benchmark performance and can be transferred to smaller VQA datasets.

  • Takeaways & Limitations

    The public YouTube-UGC version is incomplete, containing only 1147 videos, so its peer comparison is only for reference.

Abstract

from arXiv · show

Current deep video quality assessment (VQA) methods are usually with high computational costs when evaluating high-resolution videos. This cost hinders them from learning better video-quality-related representations via end-to-end training. Existing approaches typically consider naive sampling to reduce the computational cost, such as resizing and cropping. However, they obviously corrupt quality-related information in videos and are thus not optimal for learning good representations for VQA. Therefore, there is an eager need to design a new quality-retained sampling scheme for VQA. In this paper, we propose Grid Mini-patch Sampling (GMS), which allows consideration of local quality by sampling patches at their raw resolution and covers global quality with contextual relations via mini-patches sampled in uniform grids. These mini-patches are spliced and aligned temporally, named as fragments. We further build the Fragment Attention Network (FANet) specially designed to accommodate fragments as inputs. Consisting of fragments and FANet, the proposed FrAgment Sample Transformer for VQA (FAST-VQA) enables efficient end-to-end deep VQA and learns effective video-quality-related representations. It improves state-of-the-art accuracy by around 10% while reducing 99.5% FLOPs on 1080P high-resolution videos. The newly learned video-quality-related representations can also be transferred into smaller VQA datasets, boosting performance in these scenarios. Extensive experiments show that FAST-VQA has good performance on inputs of various resolutions while retaining high efficiency. We publish our code at https://github.com/timothyhtimothy/FAST-VQA.

1 Introduction

Deep VQA becomes computationally and memory intensive on high-resolution videos, while resizing and cropping can damage quality-related information. FAST-VQA addresses this with quality-retained fragments and FANet modules designed for their local and contextual information.

  • Motivation: 42.5× more FLOPs are required for 1080 × 1920 videos than for 224 × 224 inputs, limiting practical deep VQA applications.Deep VQA costs increase quadratically with resolution.
  • Motivation: Resizing corrupts local textures and cropping causes mismatched global quality, although both reduce computational cost and enable end-to-end training.These sampling artifacts can hinder learning video-quality-related representations.
  • Fragments: Grid Mini-patch Sampling uniformly samples raw-resolution mini-patches, splices them spatially, and aligns their positions temporally to form fragments.The scheme targets local textures, global coverage, and temporal variations.
  • Fragments: Fragments preserve local textures, cover global quality through uniform sampling, and retain contextual scene information while capturing temporal variations.Raw-resolution patches preserve texture details, while temporal alignment preserves inter-frame changes.
  • Fragment Attention Network: FANet is designed to extract quality information from raw-resolution patches and contextual relations while distinguishing patch discontinuities from authentic degradation.Its design follows two principles motivated by the structure of fragments.
  • Fragment Attention Network: GRPB separates intra-patch and cross-patch attention, while IP-NLR regresses mini-patch qualities independently before pooling.These modules address artificial discontinuity during attention and score regression.
  • Contributions: FAST-VQA improves accuracy by 10% over existing state-of-the-art approaches and 8% over a full-resolution Swin-T baseline with fixed recognition features.Its fragments reduce 1080P assessment complexity by 97.6% and support end-to-end learning of quality-related representations.

2 Related Works

Prior VQA work includes handcrafted methods, fixed-feature deep methods, and vision-transformer-based processing, while common VQA datasets remain small. FAST-VQA's fragment-sampling pipeline provides a related end-to-end approach for efficient quality representation learning.

  • Classical VQA Methods: Handcrafted VQA features struggle to capture complicated factors affecting video quality across diverse authentic distortions.Examples include TLVQM's combined spatial-temporal features and VIDEVAL's ensemble of handcrafted features.
  • Fixed-feature-based Deep VQA Methods: Fixed-feature deep VQA methods use pretrained networks for feature extraction and train only a feature-regression network to reduce high-resolution computational cost.Examples include ResNet-50, Inception-ResNet-V2, and features pretrained on IQA or action-recognition datasets.
  • VQA Datasets: Most common VQA datasets are too small to independently learn sufficient quality representations, whereas LSVQ contains 39,076 videos.FAST-VQA transfers quality-related features learned on large-scale LSVQ to smaller VQA datasets.
  • Vision Transformers: Vision transformers process non-overlapping image or video patches with self-attention, making patch-wise operations suitable for fragment inputs.Their patch structure naturally distinguishes mini-patch edges.
  • FAST-VQA Pipeline: FAST-VQA samples fragments through grid partition, patch sampling, patch splicing, and temporal alignment before feeding them into FANet.This pipeline connects quality-retained sampling with fragment-specific attention processing.

3 Approach

FAST-VQA samples videos into fragments that preserve local textures, global coverage, contextual relations, and temporal variations, then processes them with FANet tailored to fragment inputs.

  • Overall pipeline: FAST-VQA first converts each video into fragments through GMS, then feeds those fragments into FANet for quality prediction.GMS performs the sampling, while FANet produces the final video-quality prediction.
  • Grid Mini-patch Sampling: Uniform grid partition covers all spatial regions by dividing each frame into G_f × G_f grids of equal size.The grid uses the frame height H and width W to define spatial regions.
  • Grid Mini-patch Sampling: Raw-resolution random mini-patch sampling preserves local textures such as blurs, noise, and compression artifacts without resizing.The sampled mini-patch represents local textural quality within each grid.
  • Grid Mini-patch Sampling: Temporal alignment uses the same sampling areas across frames, retaining inter-frame temporal variations relevant to video quality.The sampling operation S is constrained to remain identical across frames for each grid.
  • Grid Mini-patch Sampling: Splicing mini-patches into their original grid positions preserves contextual relations and global scene information; the resulting samples are called fragments.Fragments are both spatially spliced and temporally aligned after GMS.
  • Fragment Attention Network (FANet): FANet adapts Swin-T to fragments using GRPB for intra- versus cross-patch attention and IP-NLR to regress before pooling.These modules address artificial discontinuities between mini-patches while extracting local and contextual quality information.

4 Experiments

The experiments define the evaluation setup around FAST-VQA variants, sampling density, resolution-specific window sizes, correlation metrics, and a differentiable PLCC loss.

  • Evaluation setup: FAST-VQA and FAST-VQA-M use different fragment sampling densities, with FANet window sizes adapted to the input sizes.FAST-VQA-M uses lower sampling density for higher efficiency.
  • Evaluation setup: PLCC and SRCC measure performance, while training uses differentiable PLCC loss l = (1−PLCC(s_pred,s_gt))^2.The implementation uses a Swin-T pretrained on Kinetics-400 and a training batch size of 16.
  • Evaluation setup: Table 2 compares FAST-VQA with FAST-VQA-M under lower sampling density using the number of grids G_f as a sampling parameter.The table caption frames the comparison around sampling density and grid count.

4.2 Benchmark Results

FAST-VQA improves VQA accuracy while substantially reducing computation relative to existing methods and the fixed-feature baseline, including on high-resolution videos.

  • At most 10% improvement over PVQ is achieved on LSVQ1080p, while FAST-VQA-M also outperforms existing state-of-the-art methods.The comparison includes classical methods, deep VQA methods, and the full-resolution Swin-T feature-regression baseline.
  • 8.10% improvement on PLCC for LSVQ1080p accompanies an end-to-end model requiring only 2.36% of the fixed-feature baseline’s FLOPs on 1080P videos.
  • The performance-FLOPs curve compares FAST-VQA with baseline methods across computational efficiency and VQA performance.
  • Up to 210× fewer FLOPs and 247× lower running time than PVQ are reported while obtaining notably better performance.

4.4 Transfer Learning with Video-quality-related Representations

FAST-VQA enables end-to-end pretraining on large VQA data and transfer to smaller datasets, where it outperforms existing state-of-the-art methods across diverse scenarios with higher efficiency.

  • FAST-VQA enables pretrain-finetune VQA by learning quality-related features end-to-end on large datasets before transfer to scenarios with limited data.
  • Across LIVE-VQC, KoNViD-1k, CVD2014, LIVE-Qualcomm, and YouTube-UGC, FAST-VQA outperforms existing state-of-the-art methods while obtaining much higher efficiency.The evaluations cover real-world, online, synthetic-distortion, selected-distortion, and user-generated video scenarios.
  • FAST-VQA performs well on YouTube-UGC videos up to 2160P, and its pretrained video-quality-related representations steadily improve performance.
  • YouTube-UGC contains 4K videos but its public version is incomplete, so peer comparisons are only for reference.

4.5 Ablation Studies on fragments

Fragment ablations show that preserving local textures, uniform global coverage, contextual relations, and temporal alignment contributes to VQA performance.

  • Comparing with resizing/cropping: The ablation study compares fragments with resizing, cropping, and alternative fragment variants while keeping the FANet structure fixed for the first part.
  • The public YouTube-UGC version is incomplete and contains only 1147 videos, limiting the peer comparison’s reference value.
  • Comparing with resizing/cropping: Fragments outperform bilinear resizing by 4% on LSVQ1080p and LIVE-VQC, and by 2% on lower-resolution scenarios.The results support retaining local textures at raw patch resolution for VQA.
  • Comparing with resizing/cropping: Fragments largely outperform both single random crops and ensembles of multiple crops, supporting uniform global quality coverage.
  • Comparing with variants of fragments: Fragments notably improve over random mini-patches and shuffled mini-patches, demonstrating the value of uniform grid partitioning and contextual relations.
  • Comparing with variants of fragments: Temporal alignment is especially beneficial on high-resolution videos, indicating that inter-frame temporal variations should be preserved.

4.6 Ablation Studies on FANet

FANet’s GRPB and IP-NLR modules address discontinuities introduced by spliced mini-patches, with improvements particularly evident for high-resolution and cross-resolution videos.

  • Effects of GRPB and IP-NLR: Both GRPB and IP-NLR produce non-negligible improvements, especially on LSVQ1080p and LIVE-VQC.
  • Effects of GRPB and IP-NLR: The corrected position biases and regression head help address problems caused by discontinuities between mini-patches.

4.7 Reliability and Robustness Analyses

FAST-VQA remains reliable despite retaining only a small fraction of each high-resolution video, and performs well across resolution groups. Single sampling is stable and nearly matches multi-sample ensemble accuracy.

  • Reliability of Single Sampling: A single fragment sampling retains stable predictions, with normalized standard deviation around 0.01 across samplings of the same video.The analysis evaluates assessment stability and relative accuracy against multiple-sample ensemble predictions.
  • Reliability of Single Sampling: 99.40% relative accuracy is achieved by one sampling compared with a 6-sample ensemble on the LSVQ1080P test set.The result holds on the pure high-resolution test set.
  • Robustness on Different Resolutions: FAST-VQA achieves ≥0.80 SRCC&PLCC across 1080P, 720P, and ≤540P LIVE-VQC resolution groups.The groups contain 110, 316, and 159 videos, respectively.

4.8 Qualitative Results: Local Quality Maps

FAST-VQA produces patch-wise local quality maps that reveal sensitivity to spatial texture changes and contextual differences between scene regions. These maps qualitatively demonstrate both local and global quality information learned from fragments.

  • Qualitative Results: Local Quality Maps: IP-NLR enables patch-wise local quality maps and re-projected frame quality maps for qualitative inspection.The maps are shown for a 1080P LIVE-VQC video.
  • Qualitative Results: Local Quality Maps: FAST-VQA distinguishes clear textures from blurry textures across frames, demonstrating sensitivity to local texture quality.The comparison includes clear Frame 0 and blurry Frames 12/24.
  • Qualitative Results: Local Quality Maps: Different predicted qualities for action-related and background areas indicate learned global scene information and contextual relations.The qualitative maps show that scene regions receive notably different quality assessments.

5 Conclusions

The paper concludes that fragments retain video-quality information more effectively than naive sampling for high-resolution VQA. FAST-VQA combines this sampling with end-to-end modeling to improve efficiency and accuracy.

  • 5 Conclusions: FAST-VQA achieves −99.5% FLOPs and +10% PLCC versus the state-of-the-art PVQ method on 1080P videos.The conclusion presents simultaneous efficiency and accuracy gains.
  • 5 Conclusions: Fragments are presented as effective VQA samples that better retain quality information than naive sampling approaches.The conclusion positions fragment sampling as a response to high-resolution computing and memory requirements.
Loading 2207.02595v1…