Source-linked AI summary
Multi-Frame Quality Enhancement for Compressed Video
Ren Yang, Mai Xu, Zulin Wang, Tianyi Li
TL;DR
Compressed-video enhancement has largely focused on individual frames despite substantial quality fluctuation and useful similarity across neighboring frames. The paper proposes MFQE, combining SVM-based PQF detection with an MF-CNN that motion-compensates adjacent PQFs before enhancing non-PQFs. Experiments report significant quality improvement, with the method outperforming the compared approaches in overall enhancement.
Problem
Existing compressed-video quality enhancement methods mainly process single frames, leaving neighboring-frame information underused despite quality fluctuation across compressed frames.
Method
MFQE detects PQFs with an SVM and uses an MF-CNN with motion compensation and quality-enhancement subnets to enhance non-PQFs from adjacent PQFs.
Results
The MFQE approach significantly improves non-PQF quality and outperforms all five compared approaches on overall quality enhancement across the test sequences.
Takeaways & Limitations
Adjacent high-quality content can be leveraged to reduce compression artifacts and mitigate quality fluctuation in compressed video.
Takeaways & Limitations
The PQF detector assumes a maximal separation between PQFs, and the parameter 2D should be adjusted for the coding standard and configurations.
Abstract
from arXiv · showhide
The past few years have witnessed great success in applying deep learning to enhance the quality of compressed image/video. The existing approaches mainly focus on enhancing the quality of a single frame, ignoring the similarity between consecutive frames. In this paper, we investigate that heavy quality fluctuation exists across compressed video frames, and thus low quality frames can be enhanced using the neighboring high quality frames, seen as Multi-Frame Quality Enhancement (MFQE). Accordingly, this paper proposes an MFQE approach for compressed video, as a first attempt in this direction. In our approach, we firstly develop a Support Vector Machine (SVM) based detector to locate Peak Quality Frames (PQFs) in compressed video. Then, a novel Multi-Frame Convolutional Neural Network (MF-CNN) is designed to enhance the quality of compressed video, in which the non-PQF and its nearest two PQFs are as the input. The MF-CNN compensates motion between the non-PQF and PQFs through the Motion Compensation subnet (MC-subnet). Subsequently, the Quality Enhancement subnet (QE-subnet) reduces compression artifacts of the non-PQF with the help of its nearest PQFs. Finally, the experiments validate the effectiveness and generality of our MFQE approach in advancing the state-of-the-art quality enhancement of compressed video. The code of our MFQE approach is available at https://github.com/ryangBUAA/MFQE.git
1. Introduction
Compressed video suffers from artifacts that degrade Quality of Experience, while existing enhancement methods largely process frames independently despite strong quality fluctuation across frames. The paper proposes MFQE, which detects Peak Quality Frames and uses them to enhance adjacent non-PQFs.
- Compression artifacts in bandwidth-constrained video can severely degrade Quality of Experience, motivating compressed-video quality enhancement.
- Single-frame enhancement methods ignore neighboring-frame information, although compressed-video quality can fluctuate dramatically across frames.
- MFQE detects PQFs with an SVM and enhances non-PQFs using a Multi-Frame CNN that takes the current frame and adjacent PQFs as inputs.
- Peak Quality Frames are frames whose quality exceeds that of their previous and subsequent frames.
- The proposed CNN-based MFQE approach reduces compression artifacts in non-PQFs by exploiting neighboring PQFs.
2. Related works
Prior compressed-video enhancement methods mainly operate on individual frames, while multi-frame super-resolution demonstrates the value of neighboring-frame information. The paper extends this multi-frame idea to compressed-video quality enhancement, where adjacent higher-quality frames can help restore low-quality frames.
- Quality enhancement: Compressed-image and compressed-video enhancement has used handcrafted methods and CNNs, but these approaches generally process already compressed content frame by frame.
- Quality enhancement: Some video enhancement methods target encoder components, whereas DCAD and DS-CNN enhance compressed video without requiring the video encoder to be modified.
- Multi-frame super-resolution: Multi-frame super-resolution methods exploit high-resolution key-frames and neighboring observations to recover information from video frames.
- Multi-frame super-resolution: For compressed video, adjacent higher-quality frames may contain content distorted in low-quality frames because heavy quality fluctuation exists across compressed frames.
- Multi-frame super-resolution: The paper presents its MFQE approach as the first attempt to apply multi-frame information for compressed-video quality enhancement.
3. Quality fluctuation of compressed video
The analysis finds substantial frame-level quality fluctuation across compressed video and shows that nearby peak-quality frames often surround lower-quality frames. Short peak separations and similar neighboring content motivate enhancing non-PQFs with information from adjacent PQFs.
- Database and measurement: The study builds a database of 70 uncompressed sequences and compresses them using MPEG-1, MPEG-2, MPEG-4, H.264/AVC, and HEVC.
- Frame-level quality fluctuation: PSNR curves across coding standards show obvious quality fluctuation along video frames.
- Frame-level quality fluctuation: All five standards have average frame-level quality STD values above 1.00 dB, with maximum STDs reaching 3.97, 4.00, 3.84, 5.67, and 3.34 dB for MPEG-1, MPEG-2, MPEG-4, H.264, and HEVC.
- PQF and VQF structure: In an HEVC example, non-PQFs, especially VQF frame 60, have lower PSNR and subjective quality than nearest PQFs at frames 57 and 61 despite similar content.
- Peak-valley relationships: Average PVD exceeds 1.00 dB in most standards, reaches 1.50 dB for HEVC, and PS is below 3 frames for H.264 and HEVC.
- Peak-valley relationships: Short PQF separations indicate that adjacent frames may share similar content, supporting enhancement of non-PQFs using information from nearest PQFs.
4. The proposed MF-CNN approach
The MFQE framework detects PQFs without reference frames, then uses neighboring PQFs to compensate motion and enhance non-PQFs. Its MF-CNN combines motion compensation with spatio-temporal quality enhancement using three frames.
- Framework: The framework first detects PQFs and non-PQFs with a no-reference detector because raw sequences are unavailable in practical enhancement.The detector refines SVM labels using PQF separation constraints.
- MC-subnet: The MC-subnet estimates temporal motion between a non-PQF and neighboring PQFs, then warps PQFs using spatial transformation and bilinear interpolation.It combines down-scaled motion estimation with additional pixel-wise motion estimation to improve motion handling without down-scaling.
- SVM-based PQF detector: The SVM detector uses 36 spatial features from each of the current, two previous, and two incoming frames, producing 180 features per frame.An RBF kernel is used, and the classifier predicts whether each frame is a PQF or non-PQF.
- MC-subnet: The MC-subnet is trained under raw-frame supervision because directly comparing distorted compressed frames would produce inaccurate motion estimates.At test time, raw frames are not required for motion compensation.
- QE-subnet: The QE-subnet receives the current non-PQF and two compensated PQFs, merging their spatial and temporal features to enhance the non-PQF.This distinguishes MF-CNN from single-frame approaches that use only within-frame spatial information.
5. Experiments
Experiments evaluate PQF detection, multi-frame quality enhancement, quality fluctuation, subjective quality, and transfer to H.264 sequences. MFQE improves compressed-video quality, especially for non-PQFs, while reducing fluctuation.
- PQF detection: At QP = 37, the SVM-based PQF detector averages 90.68% precision, 92.11% recall, and 91.09% F1-score.At QP = 42, it averages 93.98% precision, 90.86% recall, and 92.23% F1-score.
- Quality enhancement on non-PQFs: MFQE achieves larger ΔPSNR improvements for non-PQFs and VQFs than PQFs, while doubling DS-CNN’s non-PQF improvement.The comparison is reported for sequences compressed at QP = 37.
- Overall quality enhancement: At QP = 37, MFQE reaches 0.5102 dB averaged ΔPSNR, 87.78% above Li et al. and 57.86% above DS-CNN.MFQE outperforms all five compared approaches across all test sequences at QP = 37.
- Overall quality enhancement: At QP = 42, MFQE achieves 0.4610 dB ΔPSNR and doubles the second-best DS-CNN improvement of 0.2189 dB.The overall results are averaged over all frames and test sequences.
- Quality fluctuation: MFQE reduces STD and PVD relative to the HEVC baseline, whereas the five compared approaches increase both fluctuation measures.The paper links reduced fluctuation to better QoE than the compared approaches.
- Subjective quality: MFQE restores severely distorted content more effectively than the compared single-frame approaches by using neighboring high-quality frames.Examples include the mouth in Vidyo1, the ball in BasketballPass, and the shadow in PeopleOnStreet.
- Effectiveness of utilizing PQFs: Using adjacent frames instead of PQFs yields 0.3896 dB and 0.3128 dB ΔPSNR at QP = 37 and 42, versus 0.5102 dB and 0.4610 dB with PQFs.This comparison supports the effectiveness of selecting PQFs for enhancement.
- Generalization capability: Fine-tuning MF-CNN on H.264 sequences increases the average PSNR of test sequences by 0.4540 dB.The experiment evaluates transfer of MFQE to H.264-compressed sequences.
6. Conclusion
The paper proposes MFQE to reduce compression artifacts by enhancing non-PQFs with neighboring PQFs. Its experiments report stronger non-PQF and overall enhancement with less quality fluctuation than other approaches.
- Conclusion: MFQE combines an SVM-based PQF detector with an MF-CNN containing motion compensation and quality enhancement subnets.The MF-CNN uses compensated neighboring PQFs to enhance each non-PQF.
- Conclusion: MFQE significantly improves non-PQFs, producing higher overall quality enhancement and less quality fluctuation than other state-of-the-art approaches.These are the paper’s reported experimental conclusions.