PaperlayerView exact revision on arXiv

Source-linked AI summary

Bit Allocation Transfer for Perceptual Quality Enhancement of VVC Intra Coding

Runyu Yang, Ivan V. Bajić

arXiv:2510.10970v2eess.IV

TL;DR

Block-based codecs make perceptual-metric optimization difficult because such metrics do not have the additive separability of MSE. This paper transfers bit-allocation knowledge from end-to-end compression to VVC through a lightweight quantization-step model, reporting reduced generation time and competitive perceptual quality. Its LPIPS-optimized variant showed no significant advantage.

  • Problem

    MSE and PSNR correlate poorly with human perceptual quality, while metrics such as MS-SSIM and LPIPS are computationally difficult to optimize block by block.

    Source: §I · p. 1

  • Method

    A lightweight model trained with perceptual losses generates quantization steps that are converted into a bit-ratio map and then a VVC QP map.

    Source: A. Overall scheme · p. 2, §V · p. 4

  • Results

    The method reduces QP-map generation time compared with dual-encoder approaches and offers comparable or superior perceptual quality to Distillation while retaining better PSNR.

    Source: §V · pp. 4–5

  • Takeaways & Limitations

    The scheme provides a low-complexity pathway for perceptual quality enhancement in traditional codecs.

    Source: §V · p. 5

    The LPIPS-optimized variant showed no significant advantage, consistent with prior findings.

    Source: B. Results of different optimization… · p. 4

Abstract

from arXiv · show

Mainstream image and video coding standards -- including state-of-the-art codecs like H.266/VVC, AVS3, and AV1 -- adopt a block-based hybrid coding framework. While this framework facilitates straightforward optimization for Peak Signal-to-Noise Ratio (PSNR), it struggles to effectively optimize perceptually-aligned metrics such as Multi-Scale Structural Similarity (MS-SSIM). To address this challenge, this paper proposes a low-complexity method to enhance perceptual quality in VVC intra coding by transferring bit allocation knowledge from end-to-end image compression. We introduce a lightweight model trained with perceptual losses to generate a quantization step map. This map implicitly captures block-level perceptual importance, enabling efficient derivation of a QP map for VVC. Experiments on Kodak and CLIC datasets demonstrate significant advantages, both in execution time and perceptual metric performance, with more than 11% BD-rate reduction in terms of MS-SSIM. Our scheme provides an efficient, practical pathway for perceptual enhancement of traditional codecs.

I. INTRODUCTION

Block-based codecs support granular rate-distortion decisions, but conventional distortion measures poorly reflect perceptual quality and perceptual metrics are difficult to optimize block by block. The paper asks whether perceptual bit-allocation knowledge from end-to-end compression can guide traditional encoders with lower complexity.

I. INTRODUCTION

  • Block-based coding enables granular rate-distortion decisions over prediction mode, partition size, and quantization parameter for each block.These localized choices support modern codecs’ compression efficiency.Source: §I · p. 1
  • MSE and PSNR correlate poorly with human perceptual quality, while MS-SSIM and LPIPS lack the additive separability needed for practical block-by-block optimization.The resulting computational complexity limits their practical adoption in standard coding frameworks.Source: §I · p. 1
  • End-to-end compression can incorporate differentiable perceptual metrics during global training, learning spatial bit-allocation patterns that reflect relative perceptual importance.Models optimized for different metrics produce different block-wise allocation patterns.Source: §I · p. 2
  • The paper transfers learned perceptual block importance to VVC using a lightweight quantization-step model, reducing the complexity of prior approaches that run the encoder twice.Earlier transfer methods showed initial MS-SSIM gains but required two encoder runs,.Source: §I · p. 2

II. PROPOSED METHOD

The proposed scheme is introduced here, with its components described in later sections; Table I is identified as a Kodak BD-rate comparison across optimization metrics.

II. PROPOSED METHOD

  • The section introduces the proposed scheme and directs readers to later sections for detailed descriptions of its components.Source: §II · p. 2
  • Table I reports average BD-rate results for different metrics compared with an MSE-optimized NIC.Source: Table I · p. 2
  • The stated scope of Table I is the Kodak image set.Source: Table I · p. 2

A. Overall scheme

The scheme turns perceptual-loss-trained quantization steps into a rate-aligned bit-ratio map and then a QP map for VVC. Its implementation uses a model-agnostic end-to-end compression framework and describes the loss terms and latent resolution.

A. Overall scheme

  • A perceptual-loss-trained quantization-step map is reciprocated, normalized for rate alignment, and converted through the R-λ model into a QP map for VVC.The resulting map guides adaptive bit allocation for perceptual enhancement.Source: A. Overall scheme · p. 2

B. End-to-end image compression

  • The scheme is model-agnostic if the compression architecture supports block-level bit allocation; the implementation uses NIC1 with a mainstream hyper-prior design.Source: B. End-to-end image compression · p. 2
  • The end-to-end model minimizes λD + Rmain + Rhyper, where the rate terms cover latent features and hyperpriors, and its latent features are downsampled 16-fold spatially.D may be defined by MSE or perceptual metrics; hyperpriors provide side information for entropy coding.Source: B. End-to-end image compression · p. 2

C. Quantization step generation

A perceptually trained model generates a quantization-step map whose reciprocal guides block-level bit allocation and QP adaptation in VVC. The reported gains are attributed to the map’s distillation of perceptual block importance.

C. Quantization step generation

  • The model outputs positive quantization steps that regulate latent-feature magnitudes: larger steps reduce bitrates, while smaller steps increase them.Its output resolution matches the latent feature, and softplus ensures positive values.Source: C. Quantization step generation · p. 2
  • The generated quantization-step map effectively distills perceptual block-importance knowledge from the compression model.Training fixes the image-compression model and updates only the quantization-step generation model.Source: C. Quantization step generation · pp. 2–3

D. QP adaptation

  • The bit ratio is approximated by the reciprocal of the quantization step, with the cited relationship measured on 64×64 blocks using average 4×4 steps.The figure describes a linear relationship with slope 1 on Kodak and CLIC.Source: D. QP adaptation · p. 3, Figure 3 · p. 3
  • The derived block-based bit ratio guides QP adaptation through the R-λ model, which maps rate and block-varying parameters to block-wise λ and QP values.The supplied passages identify N as 3 in the VVC settings.Source: D. QP adaptation · p. 3

A. Settings

The evaluation trains quantization-step generation on LIU4K patches and compares perceptual variants against VVC-based baselines on Kodak and CLIC. It reports standard image-quality metrics and BD-rate, with LPIPS converted to dB for that calculation.

A. Settings

  • Training data comprise 607,714 randomly cropped 256×256 patches from 1,600 LIU4K images and their 2×/4× bicubic-downsampled versions.The end-to-end compression model uses pretrained MSE-optimized weights; only the quantization-step generation model is trained.Source: A. Settings · p. 3
  • Comparisons use VTM-23.0, Distillation, and PerceptQPA-enabled VTM-23.0, with a maximum QP offset of 4 and rate alignment at QP ∈{37, 32, 27, 22}.For PerceptQPA compatibility, the evaluation retains its 64×64 QP adaptation block size while replacing its block-based QP adaptation module.Source: A. Settings · p. 3
  • The three perceptual-loss variants are 1-SSIM, 1-MS-SSIM, and LPIPS, with α set to 0.02, 0.08, and 0.04, respectively.LPIPS sums feature-space distances across five convolutional layers of an AlexNet-based network.Source: A. Settings · p. 3
  • Evaluation reports RGB PSNR, SSIM, MS-SSIM, and LPIPS; LPIPS is converted to dB scale for BD-rate calculation.Source: A. Settings · p. 3
  • Evaluation uses Kodak and dimension-compliant CLIC images, with Kodak images about 0.35MP and CLIC images larger than 2MP.Kodak contains 24 images; VTM-23.0 requires dimensions divisible by 4.Source: A. Settings · p. 3

B. Results of different optimization objectives

The results favor SSIM/MS-SSIM optimization over PerceptQPA, with CLIC also exceeding Distillation, while LPIPS shows no significant advantage. More aggressive bit-ratio adaptations improve perceptual metrics at a PSNR cost, and QP-37 visuals show gains in structured regions.

B. Results of different optimization objectives

  • On CLIC, our method surpasses PerceptQPA and Distillation in SSIM, MS-SSIM, and PSNR; on Kodak, it beats PerceptQPA perceptually but trails Distillation slightly while retaining PSNR better.The CLIC result is attributed to conservative QP adaptation benefiting high-resolution images.Source: B. Results of different optimization… · p. 4
  • The LPIPS-optimized variant shows no significant advantage, consistent with a widening performance gap between block-based and end-to-end compression for deep perceptual metrics.The authors say this gap may compromise final results.Source: B. Results of different optimization… · p. 4
  • With slope size 1.2, perceptual metrics improve at the cost of PSNR, and Kodak SSIM/MS-SSIM become comparable to Distillation.This is the more aggressive QP-adaptation variant.Source: B. Results of different optimization… · p. 4
  • The reciprocal bit-ratio approximation enables more aggressive QP adaptation than the true bit ratio, leading to greater perceptual-metric gains.Source: B. Results of different optimization… · p. 4
  • At QP 37, the MS-SSIM-optimized method improves perceptual quality over fixed-QP VTM in structured regions and matches perceptual quality at lower rate in highly textured regions.Its visual quality is described as similar to PerceptQPA and Distillation.Source: B. Results of different optimization… · p. 4

C. Computational complexity

QP-map generation is faster than Distillation, taking around 20 ms on GPU or 700 ms on CPU for a Kodak image, though the time remains non-negligible in practical applications.

C. Computational complexity

IV. DISCUSSION

The discussion identifies architectural alignment, video coding, and machine-vision objectives as directions for extending the approach.

IV. DISCUSSION

  • Using end-to-end architectures better aligned with block-based coding could help bridge the LPIPS performance gap and improve knowledge transferability.Block-based structures are given as an example of closer architectural alignment.Source: §IV · p. 4
  • Extending transfer to video coding would require addressing temporal consistency, potentially with motion-guided perceptual importance across frames.Source: §IV · p. 4
  • For machine vision, task-specific losses such as object-detection accuracy could train bit allocation to prioritize regions important to downstream algorithms.Source: §IV · p. 4

V. CONCLUSION

The paper presents a low-complexity VVC intra-coding scheme that transfers bit allocation from end-to-end image compression. It reports advantages over PerceptQPA and comparable or better perceptual quality than Distillation, with better PSNR retention.

V. CONCLUSION

  • A lightweight model trained with perceptual losses generates a quantization-step map that distills block-level importance; its reciprocal approximates bit ratio for deriving the VVC QP map.Source: §V · p. 4
  • The method achieves comparable or superior perceptual quality to Distillation while offering better PSNR retention.The conclusion particularly highlights SSIM and MS-SSIM optimization as outperforming PerceptQPA.Source: §V · pp. 4–5

Related papers

Loading 2510.10970v2…