PaperlayerView exact revision on arXiv

Source-linked AI summary

Enhancing Quality for HEVC Compressed Videos

Ren Yang, Mai Xu, Tie Liu, Zulin Wang, Zhenyu Guan

arXiv:1709.06734v2cs.MM

TL;DR

Low-bit-rate HEVC videos can exhibit severe artifacts, while existing CNN approaches do not adequately model inter-coding distortions. The paper proposes separate QE-CNN models for I and P frames and a TQEO scheme for time-constrained enhancement. Experiments report effective enhancement for both frame types and validate TQEO’s control and quality performance.

  • Problem

    Low-bit-rate HEVC streams can suffer severe visual artifacts, and existing CNN-based enhancement approaches mainly handle intra-coding rather than P-frame distortions.

  • Method

    The paper uses separate QE-CNN-I and QE-CNN-P models for intra- and inter-coding distortions, then applies TQEO to maximize enhancement under a computational-time constraint.

  • Results

    The experiments show effective quality enhancement for both I and P/B frames, while TQEO is effective in controlling computational time and enhancing quality under time constraints.

  • Takeaways & Limitations

    Decoder-side QE-CNN enhancement can improve both intra- and inter-coding quality without modifying the HEVC encoder, with TQEO supporting real-time application.

Abstract

from arXiv · show

The latest High Efficiency Video Coding (HEVC) standard has been increasingly applied to generate video streams over the Internet. However, HEVC compressed videos may incur severe quality degradation, particularly at low bit-rates. Thus, it is necessary to enhance the visual quality of HEVC videos at the decoder side. To this end, this paper proposes a Quality Enhancement Convolutional Neural Network (QE-CNN) method that does not require any modification of the encoder to achieve quality enhancement for HEVC. In particular, our QE-CNN method learns QE-CNN-I and QE-CNN-P models to reduce the distortion of HEVC I and P frames, respectively. The proposed method differs from the existing CNN-based quality enhancement approaches, which only handle intra-coding distortion and are thus not suitable for P frames. Our experimental results validate that our QE-CNN method is effective in enhancing quality for both I and P frames of HEVC videos. To apply our QE-CNN method in time-constrained scenarios, we further propose a Time-constrained Quality Enhancement Optimization (TQEO) scheme. Our TQEO scheme controls the computational time of QE-CNN to meet a target, meanwhile maximizing the quality enhancement. Next, the experimental results demonstrate the effectiveness of our TQEO scheme from the aspects of time control accuracy and quality enhancement under different time constraints. Finally, we design a prototype to implement our TQEO scheme in a real-time scenario.

I. INTRODUCTION

HEVC improves coding efficiency but can produce severe low-bit-rate artifacts, motivating decoder-side enhancement. The paper proposes separate CNNs for intra- and inter-coding distortions, plus time-constrained optimization for practical deployment.

  • Motivation: HEVC saves approximately 60% bit-rate versus H.264/AVC at similar subjective quality, but low-bit-rate streams can suffer blocking, ringing, and blurring artifacts.These artifacts may severely degrade decoder-side Quality of Experience.
  • Related work: Deep-learning image-enhancement methods such as AR-CNN, D3, and TNRD motivate CNN-based restoration, while prior HEVC methods often modify encoder-side processing.The paper positions decoder-side QE-CNN as an approach applicable to existing video streams.
  • Research gap: Existing CNN-based HEVC enhancement methods primarily address intra-coding, leaving inter-coding distortions in P/B frames insufficiently handled.VRCNN only handles intra-mode coding, while DCAD uses one model trained on HEVC compressed images for both frame types.
  • Proposed method: QE-CNN learns distinct QE-CNN-I and QE-CNN-P architectures for I and P/B frames, extracting their respective distortion features without modifying the encoder.QE-CNN-P concatenates intra- and inter-coding distortion features in a uniform network.
  • Time-constrained optimization: TQEO maximizes quality enhancement under a computational-time constraint by selecting quality-enhancement operations for individual CTUs.It uses compression and pixel-domain features, models QE-CNN-I/P complexity, and can disable or replace processing on selected CTUs.
  • Extended contribution: The paper extends prior conference work with 81 training sequences, PReLU and residual learning, and a prototype for real-time quality enhancement.The extension targets improved performance for both I and P/B frames while adding time control through TQEO.

B. Related works on HEVC complexity control

Prior HEVC complexity-control methods target encoding or decoding time through encoder or decoder adjustments. The paper’s method builds on CNN-based restoration and applies it to decoded HEVC videos.

  • HEVC complexity control: Prior HEVC complexity-control approaches reduce encoding cost by skipping CTU partition computation or limiting CTU depth while targeting minimal quality loss.These methods operate at the encoder side and control complexity through frame or CTU decisions.
  • Decoder complexity control: H.264/AVC decoding complexity can be controlled by sending decoder resource demand to the encoder, which then adjusts encoder settings.This approach couples decoder-resource information with encoder configuration.
  • QE-CNN method: The proposed QE-CNN method enhances decoded HEVC videos and includes separate architectures for I frames and P/B frames.QE-CNN-I targets I frames, while QE-CNN-P targets P/B frames.
  • AR-CNN overview: AR-CNN is a four-layer end-to-end CNN that maps JPEG-compressed images to restored images through feature extraction, denoising, nonlinear mapping, and reconstruction.Its layers are jointly optimized during training without pooling or fully connected layers.
  • AR-CNN overview: For AR-CNN’s first three layers, convolution with learned weights and biases is followed by ReLU activation, while the fourth layer performs reconstruction.The configuration is summarized in Table I.

B. The proposed QE-CNN-I

QE-CNN-I is designed to enhance HEVC I-frame quality by learning intra-coding distortion from compressed image patches. Its architecture extends AR-CNN with more effective feature extraction, an additional denoising layer, and PReLU activation.

  • Purpose: QE-CNN-I targets distortion caused by HEVC intra-coding to enhance the quality of decoded I frames.The model is introduced specifically for HEVC I-frame quality enhancement.
  • Dataset: The QE-CNN-I training dataset uses 400 BSDS500 training images and 100 non-overlapping validation images.Images are encoded with HEVC all-intra mode at QP values 32, 37, 42, and 47 before patch extraction.
  • Dataset: 40×40 patches with stride 10 produce 522,000 training pairs and 34,500 validation pairs for QE-CNN-I.The patch size reflects the maximal 32×32 TU size plus 8-pixel overlap for the 9×9 convolutional filter.
  • Architecture: HEVC intra-coding has more distortion features than JPEG because it supports multiple transform sizes and 33 directional intra-picture prediction orientations.These differences motivate expanding the AR-CNN filter capacity for HEVC.
  • Architecture: At QP = 42, AR-CNN-1 improves ΔPSNR from 0.2058 dB to 0.2203 dB, while AR-CNN-2 reaches 0.2425 dB after adding one convolutional layer.AR-CNN-2 uses five convolutional layers and further denoises feature maps; adding more than one layer decreases PSNR improvement.
  • Architecture: AR-CNN-3 replaces ReLU with PReLU and is selected as the QE-CNN-I structure after improving AR-CNN-2 performance.PReLU learns the negative-input slope, helping avoid the dead-feature issue associated with ReLU.

C. The proposed QE-CNN-P

QE-CNN-P is a nine-layer convolutional network designed to handle both intra- and inter-coding distortions in HEVC P/B frames. It is trained on paired raw and encoded patches using residual learning and MSE loss.

  • Architecture: QE-CNN-P handles both HEVC intra- and inter-coding distortions to enhance P/B-frame quality.Its design targets the mixed distortion structure of inter-coded frames.
  • Dataset: The QE-CNN-P database uses 89 sequences, with 81 for training and 8 for validation.Sequences are encoded in HEVC LDP mode at QP values 22, 27, 32, 37, 42, and 47.
  • Dataset: Training uses 1,241,880 paired 40 × 40 training patches and 279,090 validation patches generated from P frames.Patches are sampled with stride 15.
  • Architecture: The network contains 9 convolutional layers, with Conv 1–4 extracting intra-coding features and Conv 5 extracting inter-coding features.Later layers combine feature maps from these stages.
  • Training procedure: QE-CNN-P uses residual learning with MSE loss, stochastic-gradient optimization, and learning-rate decay by a factor of 10 every 40 epochs.The model is first trained at QP = 42, with other QPs fine-tuned from that model.
  • Training procedure: Six models trained at QP values 22, 27, 32, 37, 42, and 47 cover frame-level QPs from 22 to 51.Hierarchical coding produces frame-level QPs up to four higher than the sequence-level QP.

IV. EXPERIMENTAL RESULTS ON QUALITY ENHANCEMENT

The experiments compare QE-CNN with AR-CNN, VRCNN, and DCAD to evaluate quality enhancement and computational time. D3 and TNRD are excluded because they are designed for JPEG compression and cannot be implemented in HEVC.

  • Evaluation design: The experiments compare QE-CNN against AR-CNN, VRCNN, and DCAD for HEVC quality enhancement.The evaluation also includes computational-time analysis.
  • Evaluation design: D3 and TNRD are excluded because their JPEG-compression prior knowledge prevents implementation in HEVC.

A. Settings

Testing uses 17 non-overlapping JCT-VC sequences encoded with HM 16.0 in LDP mode at six QP values. Whole decoded frames, rather than training patches, are fed into the fully convolutional QE-CNN models.

  • Test configuration: The test set contains 17 JCT-VC sequences encoded with HM 16.0 LDP at QP values 22, 27, 32, 37, 42, and 47.Test sequences do not overlap with training or validation sequences.
  • Test configuration: Whole frames are input during testing because QE-CNN consists entirely of convolutional layers and accepts arbitrary spatial sizes.Training uses 40 × 40 patches, but test inputs are not patch-divided.
  • Test configuration: AR-CNN is retrained on HEVC-compressed training samples for fair comparison.

B. Performance of quality enhancement

QE-CNN improves objective and subjective quality for HEVC I and P/B frames relative to the evaluated conventional and CNN-based methods. Its gains extend to RA mode, while TQEO reduces computational complexity and preserves quality advantages under time constraints.

  • I-frame quality: At QP = 42, QE-CNN-I achieves 0.3469 dB average ∆PSNR, versus 0.2503 dB for AR-CNN, 0.2693 dB for VRCNN, and 0.2511 dB for DCAD.
  • I-frame quality: QE-CNN-I significantly outperforms AR-CNN, VRCNN, and DCAD in enhancing I-frame quality across test sequences.
  • P-frame quality: At QP = 42, QE-CNN-I achieves 0.2650 dB ∆PSNR on P frames, compared with 0.1706 dB for AR-CNN and 0.1676 dB for DCAD.
  • P-frame quality: At QP = 42, QE-CNN-P reaches 0.3407 dB average Y-PSNR improvement on P frames, 28.57% higher than QE-CNN-I’s 0.2650 dB.QE-CNN-P also doubles the PSNR improvement of AR-CNN and DCAD.
  • Subjective quality: QE-CNN achieves better subjective quality than AR-CNN and DCAD on 15 of 17 test sequences, with average DMOS 44.49 versus 54.42 and 51.86.Lower DMOS indicates the reported better subjective quality in this evaluation.
  • Rate-distortion performance: The method’s BD-rate performance is evaluated against the HM 16.0 baseline using Y-PSNR results across two QP ranges.The supplied passage reports that QE-CNN saves BD-rate up to 14.
  • RA-mode quality: At QP = 32 in RA mode, QE-CNN improves B-frame ∆PSNR by 0.3065 dB, versus 0.1694 dB for DCAD and 0.1121 dB for AR-CNN.The RA results are comparable to those obtained in LDP mode.

C. Ablation experiments

The ablation experiments evaluate how network depth, activation function, and architectural updates affect QE-CNN quality enhancement. PReLU and the revised QE-CNN outperform the tested alternatives, while adding layers reduces performance.

  • Adding more layers: Adding one layer reduces ΔPSNR from 0.3469 dB to 0.3331 dB on I frames and from 0.2650 dB to 0.2153 dB on P frames.The experiment inserts a 64-filter 3×3 layer between Conv 3 and Conv 4 for QE-CNN-I.
  • Activation functions: PReLU achieves the best quality enhancement among PReLU, ReLU, and Leaky-ReLU.At QP = 42, PReLU improves averaged PSNR by 0.3469 dB on I frames and 0.3407 dB on P frames.
  • Activation functions: Replacing PReLU with ReLU lowers ΔPSNR to 0.3165 dB for I frames and 0.3002 dB for P frames at QP = 42.With Leaky-ReLU, ΔPSNR is 0.3146 dB for I frames and 0.3370 dB for P frames.
  • Comparison with DS-CNN: QE-CNN improves ΔPSNR over DS-CNN from 0.3165 dB to 0.3469 dB for I frames and from 0.3162 to 0.3407 dB for P frames at QP = 42.Similar improvements are reported at QP = 37.

D. Computational time analysis

QE-CNN provides greater Y-PSNR improvement than AR-CNN and DCAD but requires more computation. The TQEO scheme addresses this trade-off by selecting CTU-level enhancement under a computational-time constraint.

  • Computational comparison: QE-CNN performs considerably better than AR-CNN and DCAD in terms of Y-PSNR improvement.The comparison averages computational time and Y-PSNR improvement over the evaluated videos.
  • Computational comparison: QE-CNN-I requires approximately 1.53 ms per CTU and QE-CNN-P consumes 3.90 ms per CTU, versus 0.70 ms for AR-CNN and 0.64 ms for DCAD.The higher enhancement performance therefore comes with increased computational time.
  • Motivation for TQEO: CTU-level MSE-reduction RSD is approximately or greater than 100% across all 17 test sequences, indicating large variation in enhancement potential.The RSD is averaged over frames for each sequence.
  • TQEO formulation: TQEO maximizes frame MSE reduction subject to a computational-time constraint by deciding whether to disable QE-CNN-I/P or replace QE-CNN-P with QE-CNN-I on each CTU.The formulation uses predicted enhancement and modeled computational complexity for CTU-level decisions.
  • I-frame solution: For I frames, CTUs with more allocated bits receive higher enhancement priority because their enhancement gains correlate strongly with bit allocation.Spearman correlations between enhancement reduction and bit allocation are 0.85, 0.80, 0.73, and 0.65 for QP values 32, 37, 42, and 47.

C. Solution to (15) for P frames

For P frames, TQEO predicts CTU-level enhancement using MAD-ranked features and solves a constrained integer allocation problem. A branch-and-bound solution is implemented through a lookup table for practical time control.

  • P-frame modeling: QE-CNN-I and QE-CNN-P require 1.536 ms and 3.900 ms per CTU, respectively, when processing P frames.QE-CNN-P has a reported computational-time deviation of 0.120 ms.
  • P-frame enhancement prediction: P-frame enhancement reductions for both QE-CNN-I and QE-CNN-P decrease as CTU MAD decreases, with QE-CNN-P declining faster.Therefore, larger model choices are assigned to CTUs with larger MAD values.
  • Optimization: The P-frame allocation uses N1 and N2 CTU counts constrained by N1 · tn(kn = 1) + N2 · tn(kn = 2) ≤ T.N1 and N2 denote the numbers of CTUs assigned QE-CNN-I and QE-CNN-P, respectively.
  • Enhancement modeling: Second-order polynomial fitting models the two P-frame enhancement functions, with all reported R-square values above 0.99.The fitted functions use coefficients a1, a2, b1, b2, c1, and c2.
  • Optimization solution: A branch-and-bound algorithm solves the nonlinear integer optimization, after which a lookup table returns N1 and N2 for each target time.Using T/Tmax makes the lookup solution more general across devices because the runtime ratio remains approximately unchanged.
  • Optimization solution: The resulting solution controls enhancement time for both I and P frames while maximizing video-quality enhancement.The P-frame solution completes the TQEO allocation procedure for both frame types.

VI. EXPERIMENTS FOR OUR TQEO SCHEME

The TQEO experiments evaluate time-control accuracy and quality enhancement under different constraints using the HEVC test sequences. The scheme requires selected decoding information, including frame QP and I-frame CTU bit allocation.

  • Experimental setup: TQEO performance is evaluated for both control accuracy and quality enhancement under different time constraints.Experiments use the same HEVC test sequences as the earlier evaluation on an Ubuntu PC with one GeForce GTX 1080 GPU.
  • Required decoding information: For QE-CNN, the QP value of each frame is required to select the trained model used for that frame.This information is part of the decoding information used by the enhancement pipeline.
  • Device generality: Across three tested devices, tn(kn = 1)/tn(kn = 2) remains within 0.393–0.395, supporting the use of normalized target time T/Tmax.Tmax can be estimated by recording QE-CNN-P time on a few CTUs.
  • Evaluation views: The time-control experiments use averaged performance across all test sequences and also report performance for each test sequence at QP = 32.These evaluations correspond to the averaged and per-sequence experiment views.
  • Required decoding information: For TQEO on I frames, CTU bit allocation is required to sort enhancement priority under the time constraint.No other decoding information is required beyond the stated information.

A. Accuracy of computational time control

TQEO controls enhancement computation accurately under a target time while improving quality more effectively than random CTU selection. Its subjective quality remains better than comparator methods even at 50% complexity.

  • Time control accuracy: Average MAE is below 0.150% in most cases, with a maximum of 0.333%.MAE measures the error between target and actual computational time.
  • Quality enhancement under constraints: TQEO achieves higher ΔPSNR than randomly selected CTU enhancement under each tested time constraint.The scheme optimizes quality enhancement by selecting CTUs under limited computational time.
  • Subjective quality: QE-CNN at 50% complexity achieves average DMOS 49.22, better than AR-CNN at 54.42 and DCAD at 51.86.Lower DMOS indicates better subjective quality, and the result suggests partial enhancement does not significantly reduce subjective quality.

C. Real-time Implementation

The prototype applies TQEO to real-time HEVC quality enhancement while meeting computational-time targets. It achieves measurable PSNR improvement and BD-rate savings, but currently requires four high-end GPUs.

  • Real-time implementation: The prototype uses TQEO to implement real-time quality enhancement for HEVC compressed videos.It was implemented on an Ubuntu PC with four GeForce GTX 1080 GPUs.
  • Real-time performance: Actual enhancement time satisfies the real-time constraint, with average control error of 0.073 s, or 0.73% of the 10 s sequence duration.The maximum control error is 0.19 s across 10 test sequences.
  • Quality enhancement: At QP = 47, average ΔPSNR reaches 0.2458 dB for 480p@30 Hz sequences and 0.1534 dB for 720p@60 Hz sequences.For the 480p@60 Hz sequence, ΔPSNR is 0.1493 dB.
  • Quality enhancement: The prototype achieves BD-rate savings of up to 6.83%, with an average of 6.34% for 480p@30 Hz sequences.
  • Evaluation scope: Decoding time is excluded from the prototype evaluation because the cited HEVC decoder performs fast decoding.The cited decoder decodes HEVC LDP videos at 4 Gpixels/s, or 0.14 s for a 10 s 720p@60 Hz sequence.
  • Hardware requirements: Real-time enhancement currently requires four high-end GPUs, while ASIC acceleration is identified as future work.The paper notes Cambricon-X and TPU acceleration as possible alternatives.
Loading 1709.06734v2…