Source-linked AI summary
A DenseNet Based Approach for Multi-Frame In-Loop Filter in HEVC
Tianyi Li, Mai Xu, Ren Yang, Xiaoming Tao
TL;DR
HEVC compression produces artifacts, while existing learning-based in-loop filters do not exploit multiple frames. This paper introduces an RFS and DenseNet-based MIF-Net to enhance encoded frames using neighboring higher-quality frames, achieving improved rate-distortion performance over the HM baseline and other approaches.
Problem
HEVC compression produces blocking, blurring, and ringing artifacts, while existing learning-based in-loop filters do not employ multiple frames.
Method
The paper uses an RFS to select higher-quality reference frames and a DenseNet-based MIF-Net to enhance encoded frames with spatial and temporal information.
Results
−11.62% average BD-BR and 0.39dB average BD-PSNR were achieved, outperforming the HM baseline and other state-of-the-art approaches.
Takeaways & Limitations
Leveraging multiple adjacent frames provides a more effective HEVC in-loop filter than single-frame approaches within the evaluated comparisons.
Abstract
from arXiv · showhide
High efficiency video coding (HEVC) has brought outperforming efficiency for video compression. To reduce the compression artifacts of HEVC, we propose a DenseNet based approach as the in-loop filter of HEVC, which leverages multiple adjacent frames to enhance the quality of each encoded frame. Specifically, the higher-quality frames are found by a reference frame selector (RFS). Then, a deep neural network for multi-frame in-loop filter (named MIF-Net) is developed to enhance the quality of each encoded frame by utilizing the spatial information of this frame and the temporal information of its neighboring higher-quality frames. The MIF-Net is built on the recently developed DenseNet, benefiting from the improved generalization capacity and computational efficiency. Finally, experimental results verify the effectiveness of our multi-frame in-loop filter, outperforming the HM baseline and other state-of-the-art approaches.
1 Introduction
HEVC improves compression efficiency but introduces artifacts that require filtering. Existing learning-based in-loop filters do not use multiple frames, motivating a multi-frame approach that can exploit temporal information.
- HEVC compression introduces blocking, blurring, and ringing artifacts from block-wise prediction and limited-precision quantization.
- In-loop filters improve encoded-frame quality and provide higher-quality references for successive frames, potentially improving coding efficiency.
- HEVC includes DBF and SAO, while ALF was not adopted because of heavy bit-rate overhead.
- Earlier learning-based filters improved in-loop filtering but mapped or enhanced individual frames without employing multiple frames.
- The proposed MIF replaces DBF and SAO with RFS-selected reference frames and a DenseNet-based MIF-Net using spatial and temporal information.
2 Proposed MIF Approach
The proposed MIF replaces HEVC’s DBF and SAO with a multi-frame filtering framework that selects useful neighboring frames and processes them through MIF-Net or fallback IF-Net. MIF-Net aligns selected references and combines them with the unfiltered reconstructed frame using block-adaptive and DenseNet-based quality enhancement.
- MIF-Net and IF-Net: When at least M references are available, MIF-Net aligns them with the URF and feeds the aligned references and URF into a quality enhancement network.The framework uses a motion compensation network before feature extraction and synthesis.
- MIF-Net and IF-Net: When fewer than M valid references exist, IF-Net processes only the URF; the system can also retain standard DBF and SAO as a supplementary mode.The actual choice is selected from MIF-Net, IF-Net, and the standard in-loop filters.
- Reference Frame Selector: RFS examines previous encoded frames using PSNR increments and content-correlation metrics, retaining valid frames whose quality gain is positive and correlation exceeds threshold τ.Valid candidates are scored from six channel-wise quality and similarity measures before selection.
- Reference Frame Selector: RFS-Net maps each valid reference frame’s six-dimensional metric vector to a scalar potential, then selects the M highest-potential frames.The target score reflects the PSNR of a motion-compensated reference frame against the corresponding raw frame, and training uses normalized ℓ2 loss.
- MIF-Net and IF-Net: MIF-Net uses a three-path motion compensation network, block-adaptive convolution guided by CU/TU partitions, and DenseNet dense units to merge multi-frame features.The motion compensation design adds a full-scale path, six shortcuts, and PReLU activations; the network then concatenates branch features and extracts high-level features with dense units.
3 Experimental Results
Experiments evaluate the proposed in-loop filters on HEVC test sequences using objective rate-distortion and subjective visual-quality comparisons. The MIF approach achieves the strongest reported objective performance and visibly reduces distortion.
- Experimental setup: The HIF database contains 111 raw video sequences split into 83 training, 10 validation, and 18 test sequences.
- Experimental setup: The experiments incorporate all in-loop-filtering approaches into HEVC reference software HM 16.5 and evaluate them on the JCT-VC test set.
- Objective performance: −11.62% BD-BR is achieved by the MIF approach, compared with −5.03% for HM, −6.29% for the non-local adaptive loop filter, and −9.22% for RHCNN.
- Objective performance: 0.39dB BD-PSNR is achieved by MIF, exceeding 0.16dB for HM, 0.20dB for the non-local adaptive loop filter, and 0.30dB for RHCNN.
- Subjective performance: At QP = 37 on RaceHorses and PeopleOnStreet, MIF-enhanced frames show clearer horse-tail edges and fewer blocking artifacts on pedestrians than the other approaches.
4 Conclusion
The paper proposes a DenseNet-based HEVC in-loop filter that uses multiple adjacent frames through reference-frame selection and MIF-Net. Experiments report improved coding performance over HM and other state-of-the-art approaches.
- The proposed MIF approach uses an RFS to find higher-quality frames and a DenseNet-based MIF-Net to enhance each encoded frame with adjacent-frame information.
- −11.62% average BD-BR saving and 0.39dB average BD-PSNR increment are reported, outperforming the HM baseline and other state-of-the-art approaches.