Source-linked AI summary
A Convolutional Neural Network Approach for Post-Processing in HEVC Intra Coding
Yuanying Dai, Dong Liu, Feng Wu
TL;DR
Low-bit-rate lossy compression creates artifacts that degrade reconstructed visual quality, while prior CNN post-processing research focused largely on JPEG. This paper redesigns a CNN as VRCNN for HEVC intra-coding post-processing, achieving 4.6% average BD-rate reduction against the HEVC baseline and outperforming studied networks on coding efficiency, memory cost, and computational speedup.
Problem
Low-bit-rate lossy compression produces visual artifacts, while CNN-based artifact-reduction research has focused mostly on JPEG rather than HEVC.
Method
The paper redesigns a CNN for HEVC intra-coding post-processing using variable filter sizes and residue learning, trained on natural images and tested on separate standard video sequences.
Results
4.6% average luminance BD-rate reduction is achieved versus the HEVC baseline, while VRCNN outperforms AR-CNN and VDSR in bit-rate reduction, memory cost, and computational speedup.
Takeaways & Limitations
VRCNN provides an HEVC intra-coding post-processing alternative that replaces deblocking and SAO without requiring additional bits.
Takeaways & Limitations
The experiments use separate natural-image training data and standard video test sequences, and the reported scope is HEVC intra coding rather than inter coding.
Abstract
from arXiv · showhide
Lossy image and video compression algorithms yield visually annoying artifacts including blocking, blurring, and ringing, especially at low bit-rates. To reduce these artifacts, post-processing techniques have been extensively studied. Recently, inspired by the great success of convolutional neural network (CNN) in computer vision, some researches were performed on adopting CNN in post-processing, mostly for JPEG compressed images. In this paper, we present a CNN-based post-processing algorithm for High Efficiency Video Coding (HEVC), the state-of-the-art video coding standard. We redesign a Variable-filter-size Residue-learning CNN (VRCNN) to improve the performance and to accelerate network training. Experimental results show that using our VRCNN as post-processing leads to on average 4.6% bit-rate reduction compared to HEVC baseline. The VRCNN outperforms previously studied networks in achieving higher bit-rate reduction, lower memory cost, and multiplied computational speedup.
1 Introduction
Lossy compression, particularly at low bit-rates, introduces artifacts that reduce perceptual quality. This motivates CNN-based post-processing for HEVC, culminating in the proposed VRCNN for intra coding.
- Low-bit-rate lossy compression produces blocking, blurring, ringing, and other artifacts that can severely reduce reconstructed image or video quality.The paper identifies artifact reduction or removal as an important, extensively studied problem.
- HEVC uses deblocking and sample adaptive offset (SAO) for artifact reduction, with different artifact targets and signaling costs.Deblocking targets blocking artifacts without additional bits, whereas SAO addresses general compression artifacts and requires offset signaling.
- CNNs have recently achieved success in both high-level and low-level computer-vision tasks, prompting their use for compression-artifact reduction.Prior work includes AR-CNN and other networks for JPEG images, as well as SRCNN-based post-processing in HEVC.
- The paper presents a redesigned VRCNN for HEVC intra coding that integrates variable filter sizes and residue learning.The proposed network is intended to improve performance and accelerate CNN training.
- VRCNN can replace HEVC deblocking and SAO as post-processing because it reduces general compression artifacts without requiring additional bits.The paper evaluates the network using natural-image training data and standard video sequences.
2 Our Designed CNN
VRCNN redesigns the four-layer AR-CNN architecture for HEVC by matching variable filter sizes to HEVC’s variable block transforms and learning residuals. The design also reduces parameter cost while retaining a shallow network suitable for HEVC post-processing.
- AR-CNN: AR-CNN is a 4-layer fully convolutional network whose layers perform feature extraction, enhancement, mapping, and reconstruction.It has no pooling or fully connected layer, allowing output and input to share the same size under suitable boundary conditions.
- VRCNN: VRCNN retains four fully convolutional layers but redesigns AR-CNN for HEVC rather than JPEG.The network configuration is presented in Table 2 and its structure in Fig. 1.
- VRCNN: VRCNN replaces AR-CNN’s fixed filters in the second and third layers with combinations of different-sized filters to accommodate HEVC’s variable block transforms.The second layer uses 5×5 and 3×3 filters, while the third uses 3×3 and 1×1 filters; outputs are concatenated before the next layer.
- VRCNN: VRCNN is described as the first artifact-reduction network using variable filter size, with the technique restricted to selected layers for HEVC.The first and last layers retain fixed filter sizes because their feature-extraction and reconstruction roles are not affected by HEVC’s variable block-size transform.
- VRCNN: Residue learning adds the last-layer output back to the input so the CNN learns the difference between filtered output and input.The authors report that this formulation converges much faster because the input and output are largely similar.
- VRCNN: VRCNN reduces network complexity by using more filters at smaller sizes while keeping four layers, greatly reducing parameters and memory cost relative to AR-CNN and VDSR.The paper contrasts this with a 20-layer super-resolution network that uses smaller filters.
3 Training and Using VRCNN
VRCNN is trained on non-overlapping natural-image data and tested on HEVC standard sequences, then integrated into HEVC intra coding to replace deblocking and SAO.
- Training and testing: VRCNN is trained on a collection of natural images and tested on HEVC standard test sequences with no image overlap.This setup is intended to demonstrate the generalizability of the trained network.
- Training and testing: The training objective uses a loss over VRCNN outputs and original images, with parameters optimized by stochastic gradient descent and standard backpropagation.
- Training and testing: Adjustable gradient clipping accelerates training by restricting updates to [−τ/α, τ/α] and avoiding exploding gradients when the learning rate is large.In the experiments, τ is set to 0.01; as training proceeds, the learning rate becomes smaller and the clipping range becomes too large to use.
- Using VRCNN: During HEVC intra coding, deblocking and SAO are turned off, and the compressed intra frame is fed directly into trained VRCNN to produce the final reconstructed frame.VRCNN can be used in-loop or out-of-loop in the all-intra coding setting.
- Using VRCNN: VRCNN requires no additional bits while reducing general compression artifacts, and zero-padding keeps convolution outputs the same size as inputs.The paper follows zero-padding before each convolutional module for the boundary condition.
4 Experimental Results
Experiments evaluate VRCNN against HEVC post-processing and other CNNs using separate QP-specific training and standard HEVC test sequences. VRCNN improves coding efficiency and storage cost, but CNN decoding remains non-real-time on mainstream personal computers.
- Comparison with HEVC Baseline: 4.6% average luminance BD-rate reduction was achieved by VRCNN across the tested sequences versus the HEVC baseline.The RaceHorses sequence reached 7.6% luminance BD-rate reduction and 11.5% chrominance BD-rate reduction.
- Comparison with Other Networks: VRCNN suppressed all kinds of visible artifacts and produced better visual quality than the HEVC baseline in the RaceHorses example.VDSR and VRCNN eliminated most compression artifacts, while AR-CNN retained slight blocking artifacts.
- Comparison with Other Networks: VRCNN outperformed AR-CNN and slightly outperformed VDSR in BD-rate, despite using four layers like AR-CNN.The results support competitiveness of a carefully designed shallow network with variable filter size and residue learning.
- Computational complexity: VRCNN was more than 2× faster than VDSR, although CNN decoding did not meet real-time requirements on current mainstream personal computers.VRCNN was slightly slower than AR-CNN because different filter sizes complicated parallel computing.
- Memory cost: VRCNN required the lowest memory cost among the compared trained networks.Its shallower architecture and fewer parameters than AR-CNN reduce storage requirements relative to the alternatives.
5 Conclusion
VRCNN is presented for HEVC intra-coding post-processing, outperforming AR-CNN and VDSR in bit-rate reduction, memory cost, and computational speedup.
- 4.6% average luminance BD-rate reduction is achieved versus the HEVC baseline on standard test sequences.