Source-linked AI summary
MFQE 2.0: A New Approach for Multi-frame Quality Enhancement on Compressed Video
Qunliang Xing, Zhenyu Guan, Mai Xu, Ren Yang, Tie Liu, Zulin Wang
TL;DR
Compressed-video enhancement has largely focused on single frames despite heavy quality fluctuation and similarity across consecutive frames. MFQE detects PQFs with a BiLSTM and uses an MF-CNN with motion compensation and quality enhancement subnets to exploit neighboring PQFs. The paper reports improved PQF detection and video quality enhancement, including 0.562 dB average PSNR improvement on selected sequences.
Problem
Existing compressed-image and video enhancement approaches mainly process single frames, leaving neighboring-frame similarity and compressed-video quality fluctuation underused.
Method
MFQE uses a no-reference BiLSTM detector to locate PQFs, then an MF-CNN whose motion compensation and quality enhancement subnets use adjacent PQFs to enhance compressed frames.
Results
MFQE 2.0 raises average PSNR improvement from 0.455 dB to 0.562 dB, while replacing SVM with BiLSTM raises the PQF-enhancement result from 0.528 dB to 0.562 dB.
Takeaways & Limitations
Compressed-video quality enhancement can exploit higher-quality neighboring frames rather than relying only on the current frame.
Takeaways & Limitations
The work evaluates enhancement using only PSNR and SSIM, leaving other objective metrics for future work.
Abstract
from arXiv · showhide
The past few years have witnessed great success in applying deep learning to enhance the quality of compressed image/video. The existing approaches mainly focus on enhancing the quality of a single frame, not considering the similarity between consecutive frames. Since heavy fluctuation exists across compressed video frames as investigated in this paper, frame similarity can be utilized for quality enhancement of low-quality frames given their neighboring high-quality frames. This task is Multi-Frame Quality Enhancement (MFQE). Accordingly, this paper proposes an MFQE approach for compressed video, as the first attempt in this direction. In our approach, we firstly develop a Bidirectional Long Short-Term Memory (BiLSTM) based detector to locate Peak Quality Frames (PQFs) in compressed video. Then, a novel Multi-Frame Convolutional Neural Network (MF-CNN) is designed to enhance the quality of compressed video, in which the non-PQF and its nearest two PQFs are the input. In MF-CNN, motion between the non-PQF and PQFs is compensated by a motion compensation subnet. Subsequently, a quality enhancement subnet fuses the non-PQF and compensated PQFs, and then reduces the compression artifacts of the non-PQF. Also, PQF quality is enhanced in the same way. Finally, experiments validate the effectiveness and generalization ability of our MFQE approach in advancing the state-of-the-art quality enhancement of compressed video. The code is available at https://github.com/RyanXingQL/MFQEv2.0.git.
1 INTRODUCTION
Compressed video quality fluctuates substantially across frames, while existing enhancement methods mainly process single frames. MFQE exploits higher-quality neighboring PQFs through PQF detection and multi-frame convolutional enhancement.
- Motivation: Compressed video artifacts degrade Quality of Experience and can reduce classification and recognition accuracy.Prior work verifies that compression quality enhancement can improve classification and recognition performance.
- Motivation: Single-frame enhancement methods do not use neighboring-frame information, limiting their performance when compressed-frame quality fluctuates.The paper identifies heavy quality fluctuation as an opportunity to use higher-quality frames to enhance neighboring low-quality frames.
- Proposed approach: MFQE detects Peak Quality Frames with a BiLSTM-based no-reference model and enhances non-PQFs using their adjacent PQFs.PQFs are frames whose quality exceeds that of both the previous and subsequent frames.
- Proposed approach: MF-CNN combines motion compensation and quality enhancement subnets to align, fuse, and enhance information from non-PQFs and neighboring PQFs.The motion compensation subnet handles inter-frame motion, while the spatio-temporal quality enhancement subnet merges their features.
- MFQE 2.0 extensions: MFQE 2.0 enlarges the database from 70 to 160 uncompressed videos, replaces SVM with BiLSTM detection, and introduces a lightweight enhanced QE-subnet.The reported PQF-detection F1-score rises from 91.1% to 98.2%, while average PSNR improvement rises from 0.455 dB to 0.562 dB.
2 RELATED WORKS
Prior compressed-video enhancement methods and multi-frame super-resolution approaches motivate MFQE but address different objectives. MFQE transfers multi-frame information use to compressed-video quality enhancement by exploiting similar neighboring frames and quality fluctuation.
- Compressed-image enhancement: Compressed-image research uses filtering, regression fields, sparse coding, and deep learning to reduce compression artifacts.Examples include SA-DCT, Regression Tree Fields, sparse coding, AR-CNN, and DnCNN-based approaches.
- Compressed-video enhancement: Existing compressed-video methods include VRCNN and DCAD, but VRCNN is integrated into video encoding and is impractical for already compressed video.The passage distinguishes encoder-side enhancement from enhancement applied to an already compressed stream.
- Multi-frame super-resolution: Multi-frame super-resolution uses neighboring observations, motion estimation, warping, and convolutional networks to improve the current frame’s resolution.The cited approaches include recurrent networks, FlowNet-based motion estimation, spatial transformer compensation, and ESPCN.
- Multi-frame super-resolution: Multi-frame super-resolution relies on consecutive frames containing complementary observations of the same scene or object.This motivates using adjacent frames as additional information for reconstructing the current frame.
- MFQE motivation: MFQE applies the multi-frame idea to compressed video, where adjacent higher-quality frames can enhance low-quality frames because quality fluctuates heavily across frames.The paper presents its MFQE approach as the first attempt in this direction.
3 ANALYSIS OF COMPRESSED VIDEO
The database analysis establishes substantial frame-level quality fluctuation and strong similarity between neighboring compressed frames, motivating multi-frame enhancement.
- Database: 160 uncompressed video sequences spanning diverse resolutions and five compression standards were assembled for analysis.The database includes MPEG-1, MPEG-2, MPEG-4, H.264/AVC, and HEVC compressed streams.
- Frame-level quality fluctuation: PSNR curves fluctuate substantially across frames for MPEG-1, MPEG-2, MPEG-4, H.264/AVC, and HEVC.Subjective examples from HEVC-compressed Football also show varying visual quality across frames.
- Frame-level quality fluctuation: Average PSNR SD exceeds 0.87 dB for all five standards, while average PSNR PVD exceeds 1 dB except for H.264 at 0.4732 dB.Similar fluctuation patterns are reported for SSIM.
- Similarity between neighboring frames: HEVC neighboring frames have average CC values above 0.75 and CC SD values below 0.20 when separated by at most 10 frames.Comparable similarity results are reported for the other four compression standards.
- Similarity between neighboring frames: Peak Separation averages 2.0529 frames for H.264 and 2.6641 frames for HEVC, remaining considerably below 10 frames.The short separations, together with high neighboring-frame correlation, support using nearby PQFs for enhancement.
4 THE PROPOSED MFQE APPROACH
MFQE detects Peak Quality Frames without reference frames and uses neighboring PQFs to enhance non-PQFs through a multi-frame CNN.
- Framework: MFQE detects PQFs and uses the nearest previous and subsequent PQFs to enhance non-PQFs.The approach is designed for practical settings where raw sequences are unavailable.
- PQF detector: A BiLSTM detector models current and neighboring-frame features in both forward and backward directions to locate PQFs.This captures long- and short-term correlations between PQFs and non-PQFs.
- PQF detector: The detector represents each consecutive compressed frame with an output from the BiLSTM before determining PQF labels.The supplied notation defines frame sequence inputs and corresponding BiLSTM outputs.
NonPQF
The non-PQF enhancement pipeline detects and refines PQF labels, compensates motion from neighboring PQFs, and fuses multi-frame features to reconstruct enhanced frames.
- PQF detection and postprocessing: PQF labels are refined using probabilities so that only one PQF remains among competing candidates.The detector labels frames as PQFs or non-PQFs before postprocessing.
- PQF detection and postprocessing: Consecutive PQFs are removed because the PQF definition prohibits PQFs from appearing consecutively.This postprocessing step uses the detector's frame labels.
- PQF detection and postprocessing: If more than D consecutive non-PQFs occur, one frame within the gap is assigned as a PQF, with D set from the assumed maximum PQF separation.For HEVC, the average Peak Separation is 2.66 frames.
- Motion compensation: The MC-subnet estimates motion vectors and warps neighboring PQFs to compensate temporal motion relative to the non-PQF.It uses down-scaled motion estimation followed by pixel-wise motion estimation and bilinear interpolation.
- Motion compensation: The MC-subnet is trained with raw-frame supervision because compressed-frame supervision can produce inaccurate motion estimation.Its loss minimizes the MSE between compensated raw PQFs and raw non-PQFs; raw frames are unnecessary during testing.
- Quality enhancement: The QE-subnet fuses the non-PQF with compensated neighboring PQFs, extracts multi-scale features, and predicts an enhancement residual added to the non-PQF.Dense connections support feature reuse and propagation, while joint end-to-end training combines motion compensation and quality enhancement.
5 EXPERIMENTS
Experiments show that MFQE consistently improves compressed-video quality, rate-distortion performance, and quality stability across test settings. Ablations and cross-standard tests further support contributions from the BiLSTM detector, enlarged database, post-processing, and multi-frame design.
- Overall quality enhancement: At QP = 37, MFQE reaches 0.562 dB average ∆PSNR, exceeding MFQE 1.0 by 23.5% and other listed approaches by 74.5–88.0%.Its highest sequence-level ∆PSNR is 0.920 dB for PeopleOnStreet.
- Rate-distortion performance: MFQE achieves 14.06% average BD-BR reduction, compared with 8.89% for the second-best approach, DCAD.The reduction is calculated over PSNR results at QP = 22, 27, 32, 37, and 42.
- Quality fluctuation: MFQE reduces video-quality fluctuation by lowering SD and PVD, whereas five compared approaches increase both values relative to the HEVC baseline.The authors attribute this to larger PSNR gains for non-PQFs than PQFs, which narrows their quality gap.
- Ablation study: Replacing BiLSTM with SVM lowers average ∆PSNR from 0.562 dB to 0.528 dB, a 6.0% degradation attributable to the improved PQF detector.The comparison retrains and evaluates the model with the SVM-based detector.
- Ablation study: Training with the enlarged database improves average ∆PSNR from 0.533 dB to 0.562 dB compared with the previous database.The enlarged database contains 160 rather than 70 uncompressed video sequences.
- Generalization ability: A model trained on HEVC achieves 0.422 dB average PSNR improvement on H.264, while retraining on H.264 reaches 0.464 dB.These results support generalization across the two compression standards tested.
6 CONCLUSION
The paper presents MFQE as a CNN-based approach that uses neighboring PQFs to enhance compressed-video frames, with motion compensation and quality-enhancement subnets. Its future directions include perceptual metrics and encoder-side information.
- MFQE enhances compressed-video quality by using each frame’s nearest higher-quality PQFs rather than relying only on single-frame information.The approach targets compression artifacts and quality fluctuation across frames.
- The BiLSTM-based detector classifies compressed-video frames as PQFs or non-PQFs.
- MF-CNN uses an MC-subnet to compensate motion and a QE-subnet to enhance non-PQFs with the current frame and compensated PQFs.PQF quality is enhanced in the same way.
- The approach currently optimizes PSNR and SSIM, leaving perceptual quality metrics as a future direction for improving QoE.
- Future work could use encoder-side information, such as coding-unit partitions, to further improve quality enhancement performance.