PaperlayerView exact revision on arXiv

Source-linked AI summary

Reducing Complexity of HEVC: A Deep Learning Approach

Mai Xu, Tianyi Li, Zulin Wang, Xin Deng, Ren Yang, Zhenyu Guan

arXiv:1710.01218v3cs.CV

TL;DR

HEVC achieves lower bit-rates than H.264/AVC but its quad-tree CU partition makes encoding highly complex. This paper uses large-scale CU-partition databases, HCPM, ETH-CNN, and ETH-LSTM to predict intra- and inter-mode partitions, with ablations showing improved RD performance and little or modest encoding-time impact.

  • Problem

    HEVC’s brute-force RDO search over quad-tree CU partitions creates high encoding complexity despite its bit-rate benefits.

  • Method

    The paper builds large-scale intra- and inter-mode CU-partition databases and combines HCPM with early-terminated hierarchical CNN and LSTM models.

  • Results

    The approach reduces intra- and inter-mode HEVC complexity, while ablations report 3.940% BD-BR saving and 0.211dB BD-PSNR increase for the deep CNN structure.

  • Takeaways & Limitations

    Predicting CU partitions with hierarchical deep models replaces brute-force partition search within the paper’s intra- and inter-mode HEVC setting.

Abstract

from arXiv · show

High Efficiency Video Coding (HEVC) significantly reduces bit-rates over the proceeding H.264 standard but at the expense of extremely high encoding complexity. In HEVC, the quad-tree partition of coding unit (CU) consumes a large proportion of the HEVC encoding complexity, due to the bruteforce search for rate-distortion optimization (RDO). Therefore, this paper proposes a deep learning approach to predict the CU partition for reducing the HEVC complexity at both intra- and inter-modes, which is based on convolutional neural network (CNN) and long- and short-term memory (LSTM) network. First, we establish a large-scale database including substantial CU partition data for HEVC intra- and inter-modes. This enables deep learning on the CU partition. Second, we represent the CU partition of an entire coding tree unit (CTU) in the form of a hierarchical CU partition map (HCPM). Then, we propose an early-terminated hierarchical CNN (ETH-CNN) for learning to predict the HCPM. Consequently, the encoding complexity of intra-mode HEVC can be drastically reduced by replacing the brute-force search with ETH-CNN to decide the CU partition. Third, an early-terminated hierarchical LSTM (ETH-LSTM) is proposed to learn the temporal correlation of the CU partition. Then, we combine ETH-LSTM and ETH-CNN to predict the CU partition for reducing the HEVC complexity for inter-mode. Finally, experimental results show that our approach outperforms other state-of-the-art approaches in reducing the HEVC complexity at both intra- and inter-modes.

I. INTRODUCTION

HEVC lowers bit-rate substantially but incurs very high encoding complexity, motivating learned CU-partition prediction. The paper introduces databases and hierarchical CNN/LSTM models for complexity reduction in intra- and inter-mode encoding.

  • Motivation: HEVC saves approximately 50% bit-rate at similar video quality compared with H.264/AVC, but its encoding time is 253% higher on average.The complexity is attributed to advanced techniques including quad-tree CU partitioning.
  • Contributions: The paper establishes large-scale intra- and inter-mode CU-partition databases to support deep learning for HEVC complexity reduction.The database includes 2000 raw images for intra-mode and 111 raw sequences for inter-mode, each compressed at four QPs.
  • Contributions: HCPM represents an entire CTU’s structured CU partition, allowing a trained model to obtain the complete partition in one invocation.This addresses repetitive computation from predicting CU decisions separately across three CU sizes.
  • Novelty: The paper extends prior work with HCPM, early termination, inter-mode ETH-LSTM, and a large-scale inter-mode database.The conference version addressed only intra-mode complexity reduction.
  • Contributions: ETH-CNN hierarchically predicts HCPM with early termination to reduce intra-mode encoding complexity.The model automatically extracts features rather than relying on handcrafted features.
  • Contributions: ETH-LSTM learns long- and short-term CU-partition dependencies across frames for inter-mode complexity reduction.It combines with ETH-CNN features and hierarchically outputs HCPM using an early-termination mechanism.

III. CU PARTITION DATABASE

HEVC recursively partitions CTUs into variable-size CUs and selects partitions through costly RD-based RDO. The paper constructs diverse intra- and inter-mode databases containing CU split decisions from encoded images and video sequences.

  • CU Partition Overview: A CTU is recursively split by a quad-tree from the default 64×64 size to a minimum default CU size of 8×8.A CTU may contain one CU or multiple smaller CUs, with maximal CTU sizes extendable beyond 64×64.
  • CU Partition Overview: RD-cost comparison between a parent CU and its four sub-CUs determines whether the parent is split.The split flag is also included in the decision, and full RDO selects the partition with minimum RD cost.
  • CU Partition Overview: 85 possible CUs are checked in a 64×64 CTU, making recursive pre-coding the largest source of HEVC encoding time.The checked CU counts are 1, 4, 42, and 43 at sizes 64×64, 32×32, 16×16, and 8×8.
  • Intra-mode Database: The CPH-Intra database contains 2000 high-resolution images divided into training, validation, and test sets across four resolutions.The varied resolutions provide diverse training data for CU-partition prediction.
  • Intra-mode Database: 110,405,784 CU samples were collected for CPH-Intra after encoding images with HM 16.5 at four QPs.The total database contains 49.2% split-CU samples and 50.8% non-split-CU samples.
  • Inter-mode Database: The CPH-Inter database comprises 111 video sequences from 1080p, JCT-VC, and Xiph.org sources at different resolutions.Sequences were divided into non-overlapping training, validation, and test sets and encoded under LDP, LDB, and RA configurations at four QPs.

IV. COMPLEXITY REDUCTION FOR INTRA-MODE HEVC

The paper represents HEVC CU splitting hierarchically and uses HCPM to predict an entire CTU’s partition in one model invocation, avoiding repeated prediction and redundant RD checks.

  • A 64 × 64 CU can have 83,522 possible splitting patterns, making one-step multiclass prediction intractable.
  • Conventional learning methods predict splitting labels separately and invoke the model multiple times for one CTU, causing repetitive computation overhead.
  • HCPM hierarchically organizes CU-splitting labels so one model run predicts the whole CTU partition and bypasses redundant RD cost checking.

B. ETH-CNN Structure for Learning HCPM

ETH-CNN processes an entire CTU through parallel hierarchical branches and predicts HCPM labels at three levels, with early termination to skip unnecessary deeper computations.

  • ETH-CNN takes an entire CTU as input and hierarchically produces three structured output branches corresponding to HCPM levels.
  • The raw CTU is mean-removed and down-sampled in three parallel branches aligned with the three HCPM levels.
  • Each branch uses three convolutional layers, with 4 × 4 kernels first and 2 × 2 kernels thereafter, using non-overlapping strides.
  • Features from all branches are concatenated and processed by three fully connected branches whose outputs contain 1, 4 and 16 binary labels.
  • ETH-CNN supplements feature vectors with QP and uses early termination to skip deeper branches when higher-level splitting is rejected.
  • ETH-CNN has 1,287,189 trainable parameters and shares parameters while learning all structured HCPM labels, reducing prediction complexity relative to sequential approaches.

C. Loss function for training ETH-CNN model

ETH-CNN is trained with a cross-entropy loss over valid HCPM labels, optimized across training samples using stochastic gradient descent with momentum.

  • For each training sample, the loss sums cross-entropies over all valid elements of its HCPM label map.
  • Labels that do not exist in the hierarchical map are excluded from the loss, while valid level-2 and level-3 labels are counted.
  • The ETH-CNN model is trained by optimizing the loss across all training samples.
  • Stochastic gradient descent with momentum trains ETH-CNN, whose trained model predicts HEVC CU partitions as HCPM.

D. Bi-threshold Decision Scheme

The bi-threshold scheme introduces an uncertain zone between splitting and non-splitting decisions to trade encoding complexity against rate-distortion performance. Wider uncertain zones reduce misclassification but require checking more coding units.

  • Decision thresholds: Each HCPM level uses splitting threshold α_l and non-splitting threshold ᾱ_l, with ᾱ_l ≤ α_l.Predictions within [ᾱ_l, α_l] are treated as uncertain.
  • Uncertain-zone processing: Within the uncertain zone, the parent CU and its four sub-CUs are all checked using RD cost.This reduces false predictions of CU-splitting labels but adds RD evaluations.
  • Complexity–RD trade-off: Wider uncertain zones increase encoding complexity while decreasing misclassification of CU splitting.The scheme therefore exposes an explicit complexity–RD performance trade-off.
  • Threshold selection: Threshold combinations can be traversed to reduce complexity subject to an RD-degradation constraint, but no single combination fits all requirements and video sequences.Complexity reduction and RD degradation vary substantially with encoding requirements and video content.
  • Threshold selection: The adopted hierarchy constrains lower-level uncertain zones to be narrower: [ᾱ_1, α_1] ⊂ [ᾱ_2, α_2] ⊂ [ᾱ_3, α_3].This setting follows the stated complexity–RD design assumption.

E. Computational Complexity of ETH-CNN

ETH-CNN is computationally lightweight and becomes cheaper through early termination. Its measured operation count and FPGA timing support practical intra-mode complexity reduction.

  • Operation count: 3.05 × 10^6 floating-point operations are required by ETH-CNN without early termination.This total comprises 1,497,584 additions and 1,552,149 multiplications.
  • Operation count: ETH-CNN performs at least three orders of magnitude fewer floating-point operations than Alex-Net and VGG-Net.The cited comparison is approximately 3 × 10^9 operations for Alex-Net and 4 × 10^10 for VGG-Net.
  • Early termination: 12.6% of ETH-CNN floating-point operations are saved by early termination for 18 standard JCT-VC sequences at intra-mode and QP = 32.The reported average is measured during practical compression.
  • Deployment cost: For 1080p@30 Hz intra-mode sequences, the approach requires ∼40.8 G single-precision FLOPS and less than 3 ms per frame on the specified FPGA.The FPGA result is reported for the Virtex UltraScale+ VU13P.

V. COMPLEXITY REDUCTION FOR INTER-MODE HEVC

Inter-mode HEVC exploits temporal CU-partition dependencies with ETH-LSTM, which combines re-trained ETH-CNN features and hierarchical recurrent prediction to generate HCPMs with early termination.

  • Temporal correlation: CU-partition similarity is high across neighboring frames and decreases as temporal distance increases.This observation motivates modeling temporal dependencies for inter-mode prediction.
  • Temporal correlation: Across 93 sequences, CU-depth correlation remains positive while correlation coefficient decreases and mean squared error increases with frame distance.The measurements span distances from 1 to 25 GOPs and four QP values.
  • ETH-LSTM framework: ETH-LSTM learns long- and short-term CU-partition dependencies across frames using LSTM cells arranged at three HCPM levels.The levels correspond to CU depths represented by hierarchical splitting labels.
  • ETH-LSTM framework: The model precodes each current frame with CU and PU sizes forced to 64×64, feeds the residual CTU into ETH-CNN, and supplies its features to ETH-LSTM.Precoding consumes less than 3% of total encoding time and ETH-CNN is re-trained on inter-mode residuals and ground-truth splits.
  • Early termination: ETH-LSTM predicts the current frame’s CTU partition by outputting an HCPM, while early termination skips deeper calculations when higher-level splitting is rejected.The same early-termination principle is applied at subsequent levels.
  • Decision mechanism: The binarization process can apply the bi-threshold scheme to trade prediction accuracy against complexity reduction.This connects inter-mode prediction with the decision mechanism described for ETH-CNN.

C. Computational Complexity of ETH-LSTM

ETH-LSTM adds relatively few operations and gains further savings from early termination, while the combined ETH-CNN and ETH-LSTM approach remains within the reported CPU and FPGA capacity figures.

  • Operation count: 757,118 additions and 759,273 multiplications are required by ETH-LSTM, almost half the operation count of ETH-CNN.These counts are listed for the ETH-LSTM configuration.
  • Early termination: 4.3% of ETH-LSTM floating-point operations are saved by early termination for 18 standard JCT-VC sequences at QP = 32.The reported configuration uses the inter-mode setup from the experimental results.
  • Combined cost: 4.37 × 10^6 single-precision floating-point operations are consumed per CTU by the combined complexity-reduction approach.ETH-LSTM requires extracted ETH-CNN features as input.
  • Deployment cost: The combined approach requires 66.9 G single-precision FLOPS for 1080p@30 Hz sequences and less than 3 ms per frame on the specified FPGA.The cited FPGA device is the Virtex UltraScale+ VU13P.

VI. EXPERIMENTAL RESULTS

The experiments evaluate the proposed complexity-reduction approach for intra- and inter-mode HEVC using HM 16.5, standard configurations, multiple QPs, and test images and sequences.

  • Evaluation scope: Intra-mode evaluation compares the approach with SVM- and CNN-based state-of-the-art methods.Inter-mode evaluation compares it with three recent approaches.
  • Experimental setup: All approaches were implemented in HEVC reference software HM 16.5 using AI for intra-mode and LDP for inter-mode.The experiments used the default HM configuration files.
  • Experimental setup: Four QPs—22, 27, 32, and 37—were used for compression experiments.Testing covered 18 JCT-VC sequences and all 200 test images from CPH-Intra for intra-mode evaluation.
  • Experimental setup: A GeForce GTX 1080 GPU accelerated training but was disabled during complexity-reduction testing.This separates training acceleration from the reported testing runtime.
  • Training settings: ETH-CNN models were trained for intra- and inter-mode data, while ETH-LSTM was trained using the CPH-Inter training set.Hyperparameters were tuned on the corresponding validation sets.
  • Test settings: During testing, ETH-LSTM propagated hidden state across all P frames and was invoked one frame at a time.The output for frame t was calculated after processing frame t−1.

B. Performance Evaluation at Intra-mode

In intra-mode HEVC, ETH-CNN converges, predicts CU partitions accurately, and reduces encoding complexity while preserving rate-distortion performance better than competing approaches.

  • Complexity reduction: 64.01%, 66.09%, 68.11%, and 70.52% average complexity reductions are achieved at QP 22, 27, 32, and 37, respectively.These values exceed the corresponding reductions reported for and.
  • Complexity reduction: The complexity-reduction advantage over competing approaches increases as QP rises.Higher QP produces more large CUs, allowing ETH-CNN early termination to operate on more CTUs.
  • Rate-distortion performance: For images and sequences, ETH-CNN produces average BD-BR increases of 1.386% and 2.247%, respectively.Its average BD-PSNR losses are −0.081 dB for images and −0.104 dB for sequences.
  • Complexity-rate-distortion trade-off: Narrowing the uncertain zone increases the absolute values of both encoding-time savings and BD-PSNR degradation.The bi-threshold scheme therefore controls the trade-off between rate-distortion performance and complexity.

C. Performance Evaluation at Inter-mode

For inter-mode HEVC, ETH-CNN and ETH-LSTM exploit hierarchical CU maps and temporal information to achieve accurate prediction, lower complexity, and favorable rate-distortion results.

  • Prediction accuracy: ETH-LSTM predicts three CU-partition levels with average accuracies of 93.89%, 88.01%, and 80.91%.These exceed the corresponding three-level joint SVM accuracies of 90.21%, 82.62%, and 79.42%.
  • Complexity reduction: 43.84%, 52.13%, 57.89%, and 62.94% average complexity reductions are achieved at QP 22, 27, 32, and 37, respectively.These reductions exceed those of the three compared approaches at the reported QPs.
  • Complexity reduction: Predicting the entire CTU partition as an HCPM contributes to the inter-mode complexity reduction.The approach predicts all CU partitions of an entire CTU at a time.
  • Rate-distortion performance: The ETH-LSTM approach achieves 1.495% average BD-BR, compared with 1.799% for, 5.051% for, and 3.616% for.The reported BD-PSNR for the approach is −0.046 dB.
  • Runtime: ETH-CNN and ETH-LSTM run for less time than original HM at the LDP configuration.Their time proportion is lower in inter-mode because inter-mode encoding takes substantially longer than intra-mode encoding.
  • Ablation study: Residual CTUs improve rate-distortion performance by 5.461% BD-BR saving and 0.142 dB BD-PSNR increase, at a 1.12%−2.24% encoding-time cost.This comparison replaces original CTUs with residual CTUs during both training and testing.
  • Ablation study: Adding ETH-LSTM to ETH-CNN substantially improves rate-distortion performance with insignificant encoding-time change.ETH-LSTM adds little computation because it uses far fewer floating-point operations than ETH-CNN.

F. Performance Evaluation with Various Settings

Additional evaluations examine training-set size, QP generalization, coding configurations, and scene changes, while the conclusion identifies broader extensions and implementation directions.

  • Training-data size: Fig. 11 reports rate-distortion performance for the full and reduced training sets at AI and LDP configurations.The figure caption specifies the image and sequence counts for each subset.
  • Training-data size: Reduced training sets were created by randomly selecting 1/2, 1/4, and 1/8 of the full image or sequence sets, followed by model retraining.The full sets contain 1700 training images and 83 training sequences.
  • QP generalization: Models trained at QP 22, 27, 32, and 37 were directly tested at eight additional QPs without retraining.The additional QPs are 20, 24, 26, 28, 30, 34, 36, and 38.
  • Coding configurations: The approach was additionally evaluated at LDB and RA configurations using models retrained on corresponding configuration data.For RA, ETH-LSTM length was set to 32 and training samples were non-overlapping.
  • Scene changes: Experiments on sequences with drastic scene changes report that the approach outperforms other approaches and is robust on scene-changed frames.The evaluation covered three sequences under LDP and RA configurations.
  • Conclusion: The paper concludes that ETH-CNN and ETH-LSTM with HCPM and early termination outperform state-of-the-art approaches in complexity reduction and rate-distortion performance.The conclusion frames the method as replacing brute-force CU-partition RDO search.
  • Future work: Future work includes predicting PU and TU partitions, accelerating neural-network execution, and implementing the approach on FPGA devices.These directions extend beyond the paper’s current CPU-based CU-partition focus.
Loading 1710.01218v3…