Source-linked AI summary
Neural video codecs quality assessment dataset and benchmark
Nikolay Safonov, Nikita Gornostaev, Alexandra Dubonos, Dmitriy Vatolin
TL;DR
Neural video codecs introduce distinct compression artifacts and temporal strategies, exposing limitations in existing quality assessment. The paper constructs a large subjective benchmark spanning neural and traditional codecs, using crowdsourced pairwise judgments and broad metric evaluation. It concludes that current metrics behave inconsistently on neural compression artifacts, supporting dedicated NVC quality metrics.
Problem
Neural video codecs create artifacts and temporal compression effects that differ from traditional codecs, while diverse benchmarks for assessing their perceptual quality remain limited.
Method
The paper builds a 2,880-sequence dataset from neural and traditional codecs, collects crowdsourced pairwise judgments, and benchmarks full-reference and no-reference VQA metrics.
Results
Existing VQA metrics exhibit inconsistent performance on neural compression artifacts, highlighting limitations of current evaluation approaches.
Takeaways & Limitations
The dataset provides a benchmark for developing and evaluating perceptually aligned video-quality models tailored to neural video codecs.
Takeaways & Limitations
Pairwise-derived quality scores are comparable only within groups of videos sharing the same reference sequence.
Abstract
from arXiv · showhide
Video traffic constitutes a significant share of global web traffic. To reduce its volume, video codecs have been developed and continuously improved. While the industry has achieved substantial progress in traditional video coding, neural video codecs (NVCs) have recently emerged as a new approach that applies deep learning to video compression. This creates new challenges for compression quality assessment, which is essential for the further development and improvement of such codecs. In particular, it is important to evaluate the novel temporal compression paradigms introduced by NVCs. In this work, we present a large-scale subjective dataset of videos compressed with both neural and traditional video codecs. The subjective scores were collected through crowd-sourced pairwise comparisons. The proposed dataset provides a valuable resource for the development and benchmarking of video quality metrics tailored to neural video codecs. The dataset is available at the following link: https://videoprocessing.github.io/nvc-dataset-benchmark
1 Introduction
Neural video codecs introduce compression artifacts and temporal strategies that differ from traditional codecs, creating new quality-assessment challenges. The paper addresses limited benchmarking resources with a large subjective dataset and broad VQA evaluation.
- Neural video codecs use learned representations and new temporal compression strategies, producing artifacts that differ from traditional codecs.
- The paper introduces a large-scale subjective dataset containing 2,880 distorted video sequences compressed with neural and conventional codecs.
- Subjective quality is collected through crowdsourced pairwise comparisons and aggregated with the Bradley–Terry model.
- The benchmark evaluates diverse VQA metrics, including full-reference baselines and no-reference deep models, on neural and conventional codecs.
- Combining diverse artifacts, large scale, and comprehensive annotations provides a benchmark for developing and evaluating VQA metrics for neural video codecs.
2 Related works
Existing video-quality datasets trade off controlled distortions, authentic variability, pristine references, and accessibility. The paper identifies a need for a large-scale, realistic dataset specifically tailored to neural video compression.
- Legacy datasets provide controlled distortions and high-quality ground truth but have limited content diversity and omit many real-world artifacts.
- UGC datasets capture authentic distortions and diverse content but generally lack pristine references for full-reference VQA.
- Hybrid datasets combine authentic videos with synthetic degradations, yet some remain partially or fully inaccessible for benchmarking and reproducible research.
- The CLIC 2024 benchmark provides diverse neural-compression artifacts, while related work evaluates VQA metrics on neural and traditional codecs using 4K/UHD-1 content.
- These limitations motivate a large-scale realistic NVC dataset for reliable objective-metric benchmarking and perceptually aligned evaluation.
3 Dataset
The dataset is built from diverse screen-content references, multiple neural and traditional codecs, and crowdsourced pairwise judgments. Bradley–Terry modeling converts comparisons into subjective quality scores with uncertainty estimates.
- Dataset construction: The dataset contains 80 ten-second 1080p source videos and 2,880 distorted sequences generated with six neural and four traditional codecs.
- Source selection: Source videos are selected using spatial and temporal complexity descriptors and K-means clustering to cover distinct regions of the content space.
- Source selection: The reference set spans sports, gaming, natural environments, interviews, broadcasts, animation, user-generated content, water surfaces, and close-up faces.
- Distorted video generation: Six neural codecs are evaluated at four rate–distortion settings to capture varied artifacts and quality levels.
- Subjective assessment: Participants compare videos pairwise through a sequential interface with left, right, or cannot-decide responses, including hidden validation pairs.
- Subjective assessment: Each video pair receives exactly 10 valid annotations, and more than 12,000 unique participants contribute responses.
- Score estimation: The Bradley–Terry model estimates latent video-quality scores from pairwise preferences using maximum likelihood estimation.
- Score estimation: 95% confidence intervals for score differences are computed using asymptotic normality and the inverse Fisher information matrix.
4 Benchmarking
The benchmark evaluates objective metrics against subjective scores within groups sharing the same reference video. Correlations are computed per group and aggregated with Fisher Z-transform weighting.
- Evaluation protocol: Metrics are evaluated with public default implementations, while image-quality methods are averaged frame-wise and video-quality methods produce sequence-level estimates.
- Evaluation protocol: Because annotations are pairwise, quality scores are comparable only within groups of distorted versions derived from the same reference sequence.
- Correlation analysis: SROCC and KROCC are computed between objective metrics and subjective scores for each reference-sequence group.
- Correlation analysis: Overall correlations are obtained by Fisher Z-transform aggregation with contributions weighted by group size.
5 Experiments
The experiments characterize dataset diversity and benchmark objective quality metrics across neural and traditional codecs. Existing metrics show heterogeneous score behavior and generally weaker alignment with subjective judgments for neural compression.
- 5.1 Dataset diversity: The proposed dataset covers a broader and more uniformly distributed complexity space than existing datasets.It improves diversity over compression-oriented benchmarks while resembling the heterogeneous characteristics of large-scale UGC datasets.
- 5.2 Objective Metrics Benchmarking: The benchmark evaluates metric predictions using Pearson, Spearman, and Kendall correlations because scores are comparable only within each reference group.Correlations are reported for the full, neural-codec, and traditional-codec subsets.
- 5.2 Objective Metrics Benchmarking: Correlations are generally higher for traditional codecs than for neural codecs, indicating weaker alignment with human perception for neural compression artifacts.Figure 3 visualizes this comparison with neural-codec correlations on the x-axis and traditional-codec correlations on the y-axis.
- 5.2 Objective Metrics Benchmarking: PLCC values lie closer to parity across codecs than SROCC and KROCC, suggesting ranking consistency is a larger discrepancy than approximate linearity.The reported pattern indicates that metrics may preserve relative score distances while failing to capture perceptual ordering accurately.
- 5.2 Objective Metrics Benchmarking: Metric responses are heterogeneous, with some perceptual and learned metrics strongly correlated while others capture different aspects of visual quality.Correlations with subjective scores also vary significantly across metrics.
- 5.1 Dataset diversity: Metrics produce substantially different score ranges and dispersion patterns, so raw scores are not directly comparable.PSNR and SSIM have compact distributions, whereas LPIPS and DISTS span wider ranges.
6 Conclusion
The paper presents a large-scale benchmark for neural video compression quality assessment and evaluates existing metrics on neural and traditional codec outputs. Its experiments find inconsistent metric performance on neural compression artifacts, motivating dedicated metrics for neural video codecs.
- 6 Conclusion: The dataset contains 2,880 compressed video sequences generated by diverse neural and traditional codecs under varied rate–distortion settings.It combines realistic compression scenarios with large-scale subjective annotations.
- 6 Conclusion: Existing video quality metrics exhibit inconsistent performance on neural compression artifacts, highlighting limitations in current evaluation approaches.The paper states that dedicated quality metrics for neural video codecs are needed.
- 6 Conclusion: The dataset provides a benchmark for developing and evaluating more robust and perceptually aligned quality metrics for neural compression.The stated scope is video quality assessment in the context of neural video compression.
7 Ethical Considerations and Privacy
The study used paid crowdworkers to collect pairwise quality preferences and compensated them above the average local wage for the estimated completion time.
- 7 Ethical Considerations and Privacy: Paid crowdworkers provided survey responses expressing quality preferences for pairs of media content.Compensation exceeded the average local wage for the estimated completion time.