Source-linked AI summary
High-Fidelity Video Quality Assessment with VQA-Specific Saliency
Hakan Emre Gedik, Shashank Gupta, Alan Bovik
TL;DR
NR VQA must preserve original-resolution quality cues and dense temporal information, yet dense video processing is costly and preprocessing can disrupt compatibility with pretrained ViFMs. HFVQA uses multiscale fixed-size ST patches and jointly learned VQA-specific saliency; it reports SOTA benchmark performance while encoding as little as 12% of candidate patches.
Problem
NR VQA faces high computation costs while needing fine-grained spatial and temporal cues, and preprocessing that changes input structure can complicate reuse of pretrained ViFMs.
Source: §1 · pp. 1–2Method
HFVQA processes multiscale fixed-size ST patches compatible with pretrained ViFMs and jointly learns VQA-specific saliency to select quality-relevant regions before encoding.
Source: §1 · p. 2Results
Takeaways & Limitations
The framework maintains pretrained ViFM compatibility while reducing the amount of video encoded for quality assessment.
Source: §1 · p. 2, §5 · p. 8HFVQA incurs higher computational cost than efficient NR VQA methods, in part using 10 FPS rather than their 32 uniformly sampled frames at 3.2 FPS for 10-second, 30 FPS clips.
Source: §4.5 · pp. 7–8
Abstract
from arXiv · showhide
No-reference video quality assessment (NR VQA) has recently seen promising progress with deep learning. However, video data is inherently large, and processing them with deep models incurs high computational cost. This challenge is particularly acute in VQA, where preserving original-resolution cues and dense temporal information is critical for accuracy. Existing efficiency-driven preprocessing strategies, such as fragmenting, reduce computation but alter the input data distribution, limiting effective reuse of pretrained video foundation models (ViFMs). To address these challenges, we propose \textbf{H}igh-\textbf{F}idelity \textbf{V}ideo \textbf{Q}uality \textbf{A}ssessment (\textbf{HFVQA}), a framework built on fixed-size spatio-temporal (ST) patches that is fully compatible with pretrained ViFMs. HFVQA samples ST patches across multiple scales, including the original resolution, with minimal temporal subsampling to preserve low-level quality cues and semantic context. To limit computation, HFVQA introduces a lightweight auxiliary network trained end-to-end with the ViFM encoder to learn \textit{VQA-specific saliency}. Distilled directly from quality supervision, this saliency captures task-specific importance patterns, reflecting that video quality perception is dominated by a small subset of spatio-temporal regions. By combining high-fidelity spatio-temporal cues with learned, task-specific saliency, HFVQA achieves SOTA performance on standard NR VQA benchmarks while processing as little as 12\% of candidate ST patches, making high-fidelity ViFM-based VQA computationally tractable.
1. Introduction
NR VQA must preserve fine-grained quality cues and temporal information while controlling the high computational cost of dense video processing. HFVQA uses multiscale fixed-size ST patches compatible with pretrained ViFMs and learns VQA-specific saliency to reduce encoding.
1. Introduction
- VQA requires low-level visual cues as well as semantics, so spatial downsampling can obscure sharpness or noise and temporal subsampling can mask flicker or motion distortions.The input-processing challenge is balancing computational efficiency with preservation of fine-grained quality cues.Source: §1 · p. 1
- Fragmentation-based preprocessing can disrupt pretrained spatio-temporal continuity, while dense video processing incurs substantial computational cost.The paper identifies video processing as a central design challenge for practical VQA deployment.Source: §1 · pp. 1–2
- HFVQA samples fixed-size ST patches from original-resolution and resized videos, with denser temporal sampling to preserve both low-level cues and semantic context.Original-resolution patches retain quality cues, while resized variants contribute semantics; dense temporal sampling captures fine-grained temporal quality aspects.Source: §1 · p. 2
- A lightweight saliency module selects quality-relevant regions because exhaustive encoding of dense spatio-temporal patches is computationally prohibitive.Video quality perception is described as dominated by a relatively small subset of spatio-temporal regions.Source: §1 · p. 2
- The saliency module is trained jointly with the ViFM encoder under VQA supervision, producing dense importance maps that select and aggregate patch features.The learned importance patterns are tailored to VQA rather than generic visual saliency.Source: §1 · p. 2
- HFVQA achieves SOTA on standard NR VQA benchmarks while encoding as little as 12% of candidate ST patches, with largest gains on high-resolution and temporally challenging content.The framework processes fixed-size ST patches and uses learned saliency to select a reduced set before encoding.Source: §1 · p. 2
2. Related Work
Prior efficient NR VQA methods reduce computation through fragmenting and other preprocessing strategies, but these can alter inputs and complicate reuse of pretrained ViFMs. HFVQA preserves local spatio-temporal structure and learns quality-specific importance before encoding.
2. Related Work
- Earlier NR VQA methods used handcrafted NSS features, while deep-learning approaches later learned data-driven representations.Deep methods developed from off-the-shelf backbones with adapters or NSS features to end-to-end fine-tuning.Source: §2 · p. 2
- FastVQA-family fragmenting reduces spatial redundancy while retaining local structures, but can obscure semantic information; later methods add resized-video branches or views to recover context.The approaches described include DOVER, Zoom-VQA, CLIF-VQA, and MVQA.Source: §2 · p. 2
- Preprocessing strategies that alter the input distribution restrict architectural choices and complicate direct adoption of pretrained ViFMs, motivating HFVQA’s fixed-size ST patches.Directly pretraining ViFMs on such representations is described as impractical for most research settings because it is compute-intensive and often relies on proprietary datasets.Source: §2 · p. 3
- HFVQA learns VQA-specific saliency from MOS supervision, unlike generic visual saliency, which may not align with the importance of regions for VQA.ReLIQS explores learned quality-aware saliency for image quality assessment but does not model temporal dependencies in video.Source: §2 · p. 3
- KVQ and related methods estimate region importance after encoding, whereas HFVQA uses pre-encoding saliency to prune regions and reduce computation.Post-encoding importance-weighted pooling does not prune spatio-temporal regions before encoding.Source: §2 · p. 3
3. Method
HFVQA encodes fixed-size spatio-temporal patches sampled across resolutions with a pretrained video foundation model, then uses learned saliency to aggregate patch features into a quality score. Its end-to-end training combines margin-ranking and PLCC losses.
3.1. Proposed Framework
- HFVQA samples fixed-size spatio-temporal patches from original-resolution and resized video variants, independently encodes them with a pretrained ViFM, and aggregates them into a quality score.The framework combines multiscale patch representations with dense temporal information.Source: §3.1 · p. 3
- The lightweight saliency module predicts VQA-specific importance from the coarsest scale, with higher-scale saliency maps spatially upsampled for alignment with dense feature tokens.The saliency output is normalized to sum to one.Source: §3.1 · p. 3
- Saliency guides aggregation first within each patch and then across patches within each scale, before concatenated multiscale features enter a regression head for scalar quality prediction.Patch importance is based on its saliency mass.Source: §3.1 · p. 4
3.2. Training Objective
- End-to-end training combines margin-ranking and PLCC losses, with λ controlling the trade-off between the two terms.The PLCC loss uses batch means of ground-truth MOS and predictions.Source: §3.2 · p. 4
4. Experiments
HFVQA is evaluated across standard NR VQA benchmarks, with results indicating advantages on high-resolution and temporally challenging content. The experiments also examine patch-count efficiency, learned saliency, computational cost, and design ablations.
4.1. Datasets and Implementation Details
- Evaluation used five real-world SDR benchmarks; patches were sampled at three scales with 10 FPS video sampling and 16-frame temporal extent.For smaller benchmarks, results used the median over 10 random train/validation/test splits; the evaluation setup used PLCC and SRCC.Source: §4.1 · p. 5, §4.2 · p. 6
4.2. Main Results
- On LSVQ1080p, HFVQA set a new SOTA with 0.873/0.842 PLCC/SRCC, beating MVQA by 0.027/0.030.It also matched or slightly outperformed KVQ and MVQA on low- to mid-resolution benchmarks.Source: §4.2 · p. 6
- HFVQA consistently outperformed prior methods after fine-tuning, including gains over KVQ of 0.074/0.080 on LBVD and over MVQA of 0.020/0.027 on LIVE-VQC.On YouTube-UGC, it exceeded KVQ by 0.016/0.016 and MVQA by 0.018/0.018 in PLCC/SRCC.Source: §4.2 · p. 6
4.3. Compute vs. Performance Tradeoff
- With 32 of 258 candidate patches (12%), HFVQA achieved PLCC 0.873 on LSVQ1080p and 0.927 on KoNViD-1k.Performance saturated rapidly as encoded patches increased, while inference time and GFLOPs grew approximately linearly.Source: §4.3 · pp. 6–7, Figure 2 · p. 6
4.4. Qualitative Analysis of Learned Saliency
- Learned saliency highlighted semantically meaningful regions over time and was more diffuse than UNISAL’s sharper, concentrated responses.The visualization shows consecutive frames, saliency maps, and overlays alongside the UNISAL comparison.Source: §4.4 · p. 7, Figure 3 · p. 7
4.5. Efficiency of HFVQA
- HFVQA’s manageable GFLOPs and latency support consumer-grade deployment, but its cost exceeds efficient NR VQA methods in part because it uses a pretrained ViFM and 10 FPS sampling.The framework is plug-and-play with the encoder; lighter publicly available pretrained ViFM checkpoints remain limited.Source: §4.5 · pp. 7–8, Table 2 · p. 8
4.6. Ablation Studies
- Across all ablations, replacing saliency-guided top-k selection and saliency-weighted aggregation with random sampling and uniform weighting consistently degraded PLCC/SRCC by 0.010/0.011 on LSVQ1080p, 0.024/0.022 on LIVE-VQC, 0.009/0.009 on KoNViD-1k, and 0.013/0.011 on LBVD.Restricting sampling to the original-resolution scale alone reduced LSVQ1080p PLCC/SRCC by 0.039/0.041; the coarsest scale alone degraded fine-grained low-level cues.Source: §4.6 · p. 8
- Reducing sampling from 10 to 3 FPS cut computation by 72% but lowered LBVD PLCC/SRCC by 0.070/0.078, while KoNViD-1k fell only 0.002/0.002.LIVE-VQC also degraded by 0.025/0.023.Source: §4.6 · p. 8
5. Conclusion
HFVQA combines multiscale spatio-temporal patch sampling with learned saliency to preserve high-fidelity cues while reducing encoding. It reports SOTA performance while processing as little as 12% of the input video volume.
5. Conclusion
- HFVQA maintains SOTA performance while processing as little as 12% of the input video volume.The framework uses saliency-guided top-k patch selection to reduce exhaustive encoding.Source: §5 · p. 8
- Multiscale ST patch sampling preserves original-resolution and dense temporal cues while keeping the framework compatible with pretrained ViFMs.The framework preserves video data structure to transfer large-scale spatio-temporal priors to VQA.Source: §5 · p. 8
- Experiments show consistent gains from preserving local spatio-temporal structure and dense temporal sampling, particularly on high-resolution benchmarks.The framework also supports a controlled compute–performance tradeoff.Source: §5 · p. 8
Supplementary Material
Table 4 presents an HFVQA backbone ablation across three datasets, reporting PLCC/SRCC under the settings of Table 1.
Supplementary Material
- Table 4 covers an HFVQA backbone ablation on LSVQ, LSVQ1080p, and LIVE-VQC.The caption identifies the table's scope but does not provide cell values.Source: Table 4 · p. 12
- The reported metrics are PLCC/SRCC.Source: Table 4 · p. 12
- All Table 4 results follow the settings of Table 1.Source: Table 4 · p. 12
6. Further Ablations
The light-backbone ablation lowers computational cost but degrades performance relative to the VideoPrism-B configuration. HFVQA is plug-and-play across encoders, while efficient variants depend on capable lightweight video foundation models becoming available.
6.1. HFVQA with Light Backbones
- HFVQA is not tied to one encoder, and its cost is affected by the computationally heavy ViFM backbone.The ablation retrains the framework with Swin-S pretrained on ImageNet-1k and Kinetics-400, retaining the original 10 FPS sampling rate.Source: §6.1 · p. 12
- PLCC/SRCC decreases by 0.028/0.027 on LSVQtest and 0.051/0.055 on LSVQ1080p with HFVQA-Swin-S.The variant requires 1593 GFLOPs, and the authors attribute best performance to VideoPrism-B's stronger spatio-temporal representations and large-scale pretraining.Source: §6.1 · p. 12
- The authors expect more efficient HFVQA variants as lightweight yet capable video foundation models become available.Source: §6.1 · p. 12
- The authors compare this prospect with lighter, competitive alternatives that followed heavy CLIP-based models in the image domain.The passage names the MobileCLIP family and TinyCLIP after CLIP-based models.Source: §6.1 · p. 12
7. Architecture of the Saliency Decoder
The saliency module estimates dense spatio-temporal importance for VQA, using a pretrained backbone and decoder to guide patch selection. The decoder upsamples abstract features to input resolution and is optimized under MOS supervision while the saliency backbone remains frozen.
7. Architecture of the Saliency Decoder
- The saliency module predicts relative importance of dense spatio-temporal regions for VQA, a simpler task than full VQA.A lightweight backbone is used, with pretrained features considered particularly important for adapting to saliency estimation.Source: §7 · p. 12
- A lightweight saliency decoder maps abstract backbone features into a dense probability distribution over input-video regions.Source: §7 · p. 12
- The saliency backbone is pretrained Swin-T; its features are downsampled 32-fold spatially and 2-fold temporally before decoder upsampling.The decoder progressively restores the input resolution.Source: §7 · p. 12
- Five decoder stages combine bilinear spatial upsampling with residual blocks containing two depthwise 3D convolutions.An initial trilinear upsampling operation restores temporal resolution; the final saliency map is resized to align with feature grids at each patch-sampling scale.Source: §7 · p. 12
- During training, the saliency backbone stays frozen while only the decoder is optimized under MOS supervision.The stated strategy is intended to reduce destructive interference from randomly initialized layers and stabilize adaptation of pretrained features.Source: §7 · p. 12
8. Evolution of VQA-specific Saliency
The saliency maps become more structured during full fine-tuning and stabilize near convergence. Performance on LSVQtest improves across training stages.
8. Evolution of VQA-specific Saliency
- Training first optimizes adapter modules with pretrained backbones frozen, then jointly fine-tunes the saliency backbone, ViFM encoder, decoder, and prediction head.The adapter-only stage includes the saliency decoder and prediction head.Source: §8 · pp. 12–13
- Figure 4 traces saliency maps from adapter-only training through early and mid fine-tuning to convergence, alongside the consecutive input frames.After adapter-only training, maps are diffuse; as the saliency backbone adapts, they become more structured and discriminative.Source: §8 · p. 13, Figure 4 · p. 13
- LSVQtest SRCC rises from 0.852 after adapter-only training to 0.891 early and 0.900 mid fine-tuning, reaching 0.903 after convergence.The maps change little after mid training, suggesting stable behavior rather than erratic epoch-to-epoch fluctuation.Source: §8 · p. 13
- The converged patterns are interpreted as task-driven importance learned from MOS supervision, with some characteristics shared with conventional visual saliency and clear differences.The authors attribute the differences to task-specific characteristics induced by VQA supervision.Source: §8 · p. 13
9. Additional Qualitative Results
Qualitative examples show that learned saliency focus varies with scene content and visual conditions. The observations support an interpretation of task-relevant patterns rather than generic visual saliency.
9. Additional Qualitative Results
- Learned saliency frequently emphasizes semantically meaningful regions such as humans, animals, and prominent objects likely to influence perceived video quality.Figures 5 and 6 show input frames, saliency maps, overlays, and comparisons with UNISAL.Source: §9 · p. 13, Figure 5 · p. 14, Figure 6 · p. 15
- Across consecutive frames, saliency appears relatively consistent over time and often remains centered on dynamic scene elements, suggesting a possible bias toward temporally salient regions.The authors caution that this tendency is difficult to fully assess from still visualizations.Source: §9 · p. 13
- In low-light or low-quality scenes with weak semantic structure, saliency becomes more spatially diffuse and often concentrates around bright or visually distinct regions.The authors suggest the model may rely more on basic visual structures that can reveal compression artifacts, noise, or other distortions.Source: §9 · p. 13
- Together, the observations support the interpretation that saliency adapts its spatial focus to scene semantics and local visual characteristics rather than reflecting generic visual saliency.The section introduces additional qualitative visualizations in Figures 5 and 6.Source: §9 · p. 13