PaperlayerView exact revision on arXiv

Source-linked AI summary

BVI-DVC: A Training Database for Deep Video Compression

Di Ma, Fan Zhang, David R. Bull

arXiv:2003.13552v2eess.IVcs.CV

TL;DR

Existing video-coding training databases are often repurposed from other tasks and have limited coverage of resolutions, textures, and bit depths. This paper introduces BVI-DVC, evaluates it across four coding tools and ten CNN architectures, and reports greater coding gains than three commonly used databases under identical configurations.

  • Problem

    Publicly available databases specifically designed for learning-based video coding are lacking, while commonly used alternatives may have limited coverage of high-resolution, high-bit-depth, and diverse video content.

  • Method

    The paper proposes BVI-DVC, an 800-sequence video database spanning 270p to 2160p, and evaluates it with four video coding tools and ten CNN architectures under identical training and evaluation configurations.

  • Results

    BVI-DVC produces significantly greater coding gains than DIV2K, REDS, and CD across the tested networks and coding modules for both PSNR and VMAF, with over 4.2% and 5.4% additional bitrate savings, respectively.

  • Takeaways & Limitations

    BVI-DVC provides a publicly available training resource whose diverse content supports CNN-based video compression evaluation and may also support related image and video processing tasks.

Abstract

from arXiv · show

Deep learning methods are increasingly being applied in the optimisation of video compression algorithms and can achieve significantly enhanced coding gains, compared to conventional approaches. Such approaches often employ Convolutional Neural Networks (CNNs) which are trained on databases with relatively limited content coverage. In this paper, a new extensive and representative video database, BVI-DVC, is presented for training CNN-based video compression systems, with specific emphasis on machine learning tools that enhance conventional coding architectures, including spatial resolution and bit depth up-sampling, post-processing and in-loop filtering. BVI-DVC contains 800 sequences at various spatial resolutions from 270p to 2160p and has been evaluated on ten existing network architectures for four different coding tools. Experimental results show that this database produces significant improvements in terms of coding gains over three existing (commonly used) image/video training databases under the same training and evaluation configurations. The overall additional coding improvements by using the proposed database for all tested coding modules and CNN architectures are up to 10.3% based on the assessment of PSNR and 8.1% based on VMAF.

I. INTRODUCTION

Deep-learning video compression requires larger and more diverse training material than existing databases provide. BVI-DVC addresses this gap with a public database and systematic evaluations across coding tools and CNN architectures.

  • I. INTRODUCTION: Learning-based compression requires substantially more training material and broader content coverage than conventional compression methods.Existing databases were mainly designed for computer-vision applications such as super-resolution.
  • I. INTRODUCTION: Insufficient content coverage and diversity can prevent network generalisation and limit the performance of CNN-based video coding methods.
  • I. INTRODUCTION: BVI-DVC provides 800 progressive-scanned clips from 270p to 2160p for training CNN-based tools that enhance conventional compression algorithms.The evaluated tools include spatial resolution re-sampling, bit-depth re-sampling, post-processing and in-loop filtering.
  • I. INTRODUCTION: The study trains ten CNN architectures with BVI-DVC and compares them with three commonly used databases across four video coding tools.The resulting tools are integrated into HEVC HM 16.20 and evaluated under JVET Common Test Conditions.
  • I. INTRODUCTION: Video compression remains central to delivering increasingly demanding video content at suitable quality within transmission-bandwidth constraints.

A. Machine learning based compression

Deep neural networks have improved multiple video-coding components, but training databases remain largely repurposed from computer-vision tasks and lack suitable coding-oriented coverage.

  • A. Machine learning based compression: Deep neural networks have been applied to intra prediction, motion estimation, transforms, quantisation, entropy coding, post-processing and loop filtering.Related work also includes format adaptation and virtual reference-frame optimisation.
  • A. Machine learning based compression: No publicly available database was specifically designed for learning-based video coding, despite training data being critical for generalisation and avoiding over-fitting.
  • A. Machine learning based compression: ImageNet, DIV2K and BSDS were created primarily for object recognition, super-resolution or segmentation rather than video coding.BSDS also contains fewer, lower-resolution source images than DIV2K.
  • A. Machine learning based compression: Vimeo was designed for optical-flow and temporal-super-resolution training, contains sequences up to 448×256, and excludes dynamic textures.
  • A. Machine learning based compression: CD and REDS provide video material for super-resolution, but CD contains only 29 sequences while REDS contains 300 clips at 1280×720.
  • A. Machine learning based compression: Existing databases lack high-resolution, high-bit-depth content and generally omit dynamic textures needed by modern video coding.UCF101, for example, consists of 320×240 action videos and was designed for human action recognition.

III. THE BVI-DVC VIDEO TRAINING DATABASE

BVI-DVC is constructed from diverse UHD source material and systematically resampled to provide 800 sequences across four resolutions. Feature distributions are used to assess its content diversity.

  • III. THE BVI-DVC VIDEO TRAINING DATABASE: Two hundred source sequences were selected from public databases and cover natural scenes, objects and multiple texture types.The source collection includes static and dynamic textures, structural content and luminance-plain content.
  • III. THE BVI-DVC VIDEO TRAINING DATABASE: Figure 1 presents sample frames from 20 example BVI-DVC sequences.
  • III. THE BVI-DVC VIDEO TRAINING DATABASE: The source videos are progressive-scanned 3840×2160 clips at 24–120 fps and 10-bit YCbCr 4:2:0 format.
  • III. THE BVI-DVC VIDEO TRAINING DATABASE: Each clip is truncated to 64 frames without scene cuts, then down-sampled to 1920×1080, 960×540 and 480×270 using a third-order Lanczos filter.This produces 800 sequences at four spatial resolutions.
  • III. THE BVI-DVC VIDEO TRAINING DATABASE: BVI-DVC has relatively wide coverage of spatial information, temporal information and colourfulness across its 200 UHD source sequences.These feature distributions indicate diversity in the proposed database.

IV. EXPERIMENTS

The experiments evaluate BVI-DVC against selected image and video databases using four CNN-based coding modules and ten network architectures.

  • IV. EXPERIMENTS: Ten network architectures are evaluated with post-processing, in-loop filtering, spatial resolution adaptation and effective bit-depth adaptation.
  • IV. EXPERIMENTS: The four coding modules were selected because prior work showed significant gains over standardised codecs and stronger performance than existing end-to-end solutions.
  • IV. EXPERIMENTS: BVI-DVC is benchmarked against DIV2K, REDS and CD under the study’s available time and resource constraints.DIV2K was selected for its use in JVET contributions and CNN-based coding, while REDS and CD provide relatively higher-resolution video content.

A. Coding Modules

CNN-based post processing operates on decoded frames at the decoder to reduce compression artefacts and enhance video quality. The workflow uses the network to produce the final reconstructed frame in the same format.

  • Post processing applies decoder-side processing to reconstructed video frames to reduce compression artefacts and enhance video quality.
  • A CNN-based post-processing network takes each decoded frame as input and outputs the final reconstructed frame with the same format.
  • Figure 3 presents the coding workflow for a CNN-based post-processing module.

2) Coding Module 2 (In-loop Filtering - ILF):

In-loop filtering processes reconstructed frames at both encoder and decoder, with filtered outputs available as references for subsequent coding. The cited workflow contrasts this module with spatial-resolution adaptation, whose CNN reconstructs full-resolution frames after down-sampling.

  • In-loop Filtering (ILF): In-loop filtering processes reconstructed frames at both the encoder and decoder, and its output can serve as a reference for further encoding and decoding.
  • In-loop Filtering (ILF): The CNN-based in-loop filtering module uses the same input and output formats as CNN-based post processing.
  • In-loop Filtering (ILF): Figure 4 shows the CNN-based in-loop filtering workflow, with dashed-box modules forming the corresponding decoder.
  • Spatial Resolution Adaptation (SRA): Spatial resolution adaptation downsamples original frames by a fixed factor of 2 using Lanczos3 before encoding.
  • Spatial Resolution Adaptation (SRA): At decoding, the SRA module applies nearest-neighbour filtering before CNN-based super-resolution reconstructs full-resolution frames.

4) Coding Module 4 (Effective Bit Depth Adaptation - EBDA):

Effective bit depth adaptation changes the bit depth used during encoding and restores decoded frames to 10 bits with a CNN. The evaluation uses ten CNN architectures under identical training interfaces and includes diverse architectural designs.

  • Effective Bit Depth Adaptation (EBDA): Effective Bit Depth (EBD) is the actual bit depth representing video content, whereas Coding Bit Depth (CBD) denotes the pixel bit depth used by the encoder.
  • Effective Bit Depth Adaptation (EBDA): The experiment fixes CBD at 10 bits and reduces original frames by 1 bit through bit-shifting before CNN-based up-sampling at the decoder.
  • Employed CNN Models: Ten popular network architectures are implemented for the four coding modules, including residual-block, residual-dense-block, and non-residual designs.
  • Employed CNN Models: SRCNN is a three-convolutional-layer CNN originally designed for single-image super-resolution and trained with mean-squared-error loss.
  • Employed CNN Models: The evaluated CNNs retain their published architectures, while input and output interfaces are modified for the relevant coding formats.

C. Training Data

The training-data benchmark compares BVI-DVC with three existing databases after preparing content for the four coding tools. Training examples are formed from reconstructed and original frames sampled consistently and divided into augmented image blocks.

  • DIV2K, REDS, and CD are selected as benchmark databases for evaluating BVI-DVC training effectiveness.
  • Benchmark content is spatially down-sampled by a factor of 2 or reduced by 1 bit through bit-shifting for the corresponding coding tools.
  • Training uses original content for PP and ILF, spatially down-sampled clips for SRA, and bit-depth-reduced sequences for EBDA.
  • Frames from reconstructed sequences and their originals are randomly sampled with matching spatial and temporal rates, split into 96×96 YCbCr 4:4:4 blocks, and rotated for augmentation.

D. Network Training and Evaluation

The evaluation trains CNN models across multiple databases and configurations, then processes decoded frames in overlapping blocks before reconstructing the output frame.

  • Training configuration: 480 CNN models span four training databases, three input content groups, four QP sub-groups, and ten network architectures.Training used Adam optimisation for 200 epochs with batch size 16 and a learning rate of 0.0001.
  • Evaluation workflow: Evaluation segments decoded frames into 96×96 overlapping blocks after matching their spatial resolution.The blocks use 4-pixel overlap and YCbCr 4:4:4 input conversion.
  • Reported measurements: The PP results table reports average BD-rate percentages for ten architectures and four training databases over nineteen JVET CTC sequences, assessed with PSNR or VMAF.
  • Evaluation workflow: Output blocks are aggregated using the same pattern and converted to YCbCr 4:2:0 to form the final reconstructed frame.

V. RESULTS AND DISCUSSIONS

The study evaluates four CNN-based coding modules within HEVC using unseen JVET test sequences and BD-rate measurements under PSNR and VMAF. Across architectures and modules, BVI-DVC achieves greater coding gains than the three benchmark databases.

  • Evaluation setup: Four coding modules were integrated into HEVC HM 16.20 and tested under the JVET CTC Random Access Main10 configuration.Evaluation used nineteen SDR sequences from resolution classes A1, A2, B, C, and D, with none included in the training databases.
  • Evaluation setup: Coding performance was benchmarked against HEVC HM 16.20 using BD-rate under PSNR and VMAF quality metrics.BD-rate measures bitrate savings across the tested QP range at equivalent video quality.
  • Database comparison: BVI-DVC produced significantly greater coding gains than DIV2K, REDS, and CD for every tested architecture and coding module under both PSNR and VMAF.
  • Database comparison: 4.2% and 5.4% additional bitrate savings were obtained with BVI-DVC compared with the other databases under PSNR and VMAF, respectively.These means aggregate ten networks and four coding modules.

B. Comparison of Networks

The paper compares CNN architectures under matched training and evaluation conditions while reporting database-based coding results. Residual-block architectures perform best, and BVI-DVC is presented as a publicly available, broadly useful training resource.

  • B. Comparison of Networks: RCAN, RDN, ESRResNet, and MSRResNet outperform the other six evaluated architectures for both PSNR and VMAF.The authors associate this performance with their residual block structure.
  • B. Comparison of Networks: VDSR, FSRCNN, and SRCNN show relatively low coding gains, including coding loss under PSNR when trained on CD.The paper attributes this pattern to simpler architectures or the absence of residual learning, which may reduce training and evaluation stability.
  • Database and module evaluation: Table III reports average BD-rate percentages for ILF across ten network architectures and four training databases over nineteen JVET CTC sequences, assessed by PSNR or VMAF.
  • Database and module evaluation: Figure 7 compares average coding gains for four coding modules across ten architectures trained on BVI-DVC, DIV2K, REDS, and CD within HEVC HM 16.20.
  • Conclusion: BVI-DVC is publicly available and was evaluated with four coding modules and ten typical CNN architectures.The conclusion also identifies potential use in denoising, frame interpolation, and super-resolution, while future databases should include more immersive formats.
Loading 2003.13552v2…