PaperlayerView exact revision on arXiv

Source-linked AI summary

CASR: Content-Adaptive Neural Super-Resolution Post-Filter for Versatile Video Coding via Low-Rank Overfitting

Khoa Pham-Dinh, Francesco Cricri, Maria Santamaria, Honglei Zhang, Hamed R. Tavakoli, Moncef Gabbouj, Juho Kannala, Miska M. Hannuksela

arXiv:2609.37328v1eess.IV

TL;DR

Generic offline-trained SR models adapt poorly to diverse video content and compression artifacts. CASR performs encoder-side, per-sequence LoRA adaptation of a VVC SR post-filter, and achieves BD-rate savings over VTM21 across components and coding configurations. The experiments select rank 4 as the trade-off between coding gain and signalling cost.

Abstract

from arXiv · show

The use of super-resolution as a post-processing step following a video codec allows videos to be encoded at reduced spatial resolution at the encoder side and to be reconstructed and upsampled to the original resolution at the decoder side. In this way, the required bitrate is reduced and the quality of the reconstructed frames is improved without modifying the core coding architecture. However, generic SR models are typically trained offline and lack sufficient adaptability to the diverse content characteristics and compression artifacts produced by video codecs, which limits their effectiveness during test time. To address the limitation, this paper proposes CASR (Content-Adaptive Super-Resolution), a content-adaptive SR post-filter framework for Versatile Video Coding (VVC), based on encoder-side overfitting on each input sequence. In order to limit the bitrate overhead required for signalling the content adaptation signal, i.e. the weight-update, Low-Rank Adaptation (LoRA) is leveraged. The method freezes the convolution kernels of a pretrained SR network and fine-tunes only lightweight rank-r matrices attached to selected convolution layers, using VVC decoded frames and quantization-parameter (QP) maps of test sequences as supervision. The resulting low-rank update is compressed with the MPEG Neural Network Compression and Representation (NNR) standard. Experiments on the JVET common test conditions (CTC) class A1 and A2 sequences indicate that LoRA-based content adaptation provides bitrate savings over a non-adapted SR post-filter at a small signalling cost. Compared with the VVC Test Model (VTM21), the proposed method achieves BD-rate savings of -10.93% (Y), -15.39% (U), -24.41% (V) under random access and -13.43% (Y), -5.75% (U), -22.94% (V) under all-intra. An ablation of the LoRA rank r further shows that r=4 provides the best trade-off between coding gain and signalling cost.

I. INTRODUCTION

VVC post-processing can improve decoded pictures without changing prediction or bitstream syntax, but generic neural post-filters adapt poorly to varied content and codec artifacts. CASR addresses this gap with a VSEI-compatible, content-adaptive super-resolution post-filter using LoRA updates.

I. INTRODUCTION

  • Generic neural post-filters are commonly trained offline and applied uniformly, while content-adaptive methods specialize filtering networks to target sequences.Prior approaches include updating selected biases or multipliers, and NNR supports compression of neural-network updates.Source: §I · p. 1
  • Super-resolution post-processing can encode video at reduced spatial resolution and reconstruct it at higher resolution after decoding, improving coding efficiency and visual quality.The paper situates this approach among standalone post-filters and hybrid coding pipelines.Source: §I · p. 1
  • VSEI-compatible super-resolution post-filtering must perform upsampling within the model rather than rely on the non-normative RPR upsampler.The NNVC approach uses RPR for resolution change and the neural network only to refine resampled pictures.Source: §I · p. 2
  • CASR integrates ×2 upsampling into a VVC super-resolution post-filter whose interface is compatible with VSEI NNPF signalling.The encoder fine-tunes low-rank matrices on decoded test frames while freezing pretrained convolution weights, then compresses the update with NNR.Source: §I · p. 2

II. METHODOLOGY

The proposed pipeline downsamples each HR sequence by two in both spatial dimensions, encodes it with VTM, and restores resolution using a decoder-side SR post-filter. Encoder-side sequence adaptation is transmitted as a compressed update for that filter.

II. METHODOLOGY

  • The input HR sequence is downsampled by a factor of two in both spatial dimensions and encoded with VTM 21.0 under RA or AI configuration.The SR post-filter operates on decoded LR frames outside the coding loop.Source: §II · p. 2
  • At the encoder, per-sequence overfitting trains LoRA or multiplier matrices attached to convolutions while keeping pretrained convolution kernels frozen.The resulting update is quantized, entropy-encoded with NNR, and transmitted as side information.Source: §II · p. 2
  • At the decoder, the NNR bitstream is decoded, the update is merged into the frozen base weights, and the adapted SR network upsamples the frames.Source: §II · p. 2

A. Network architecture

The base network performs ×2 super-resolution using a residual network that adds a learned HR residual to a deterministic upsampling of the decoded picture. It processes decoded YUV and a QP map through separate luma and chroma paths.

A. Network architecture

  • The LR input comprises a decoded YUV 4:2:0 frame and a per-slice QP map, and the network performs ×2 upscaling.It maps the LR input to an HR residual added to deterministic upsampling of the decoded picture.Source: A. Network architecture · p. 2
  • The architecture packs decoded YUV into a six-channel LR tensor, combines frame and QP features, then uses sub-pixel convolution for ×2 upscaling.Source: A. Network architecture · p. 2
  • After upscaling, the network splits features into luma and chroma paths with eight and four residual blocks, respectively, before producing HR residuals.The model requires 79.4k MACs per pixel and has 137,091 trainable parameters.Source: A. Network architecture · p. 2

B. Content adaptation

CASR adapts a pretrained SR network per sequence by training lightweight parameters attached to selected convolution layers while freezing the base network weights. LoRA generalizes multiplier-only adaptation to low-rank updates of convolution kernels.

B. Content adaptation

  • LoRA adapts selected convolution kernels with two trainable low-rank matrices per layer while freezing pretrained kernels, biases, and PReLU coefficients.The multiplier baseline instead trains channel-wise vectors that scale post-activation outputs.Source: B. Content adaptation · p. 2
  • A channel-wise multiplier is a rank-1 update of a 1×1 kernel, whereas LoRA permits rank-r updates to every kernel shape used in the network.The LoRA rank is r and α is its scaling factor.Source: B. Content adaptation · p. 2
  • LoRA parameters remain a small fraction of the base model, and the adapted update is compressed for transmission with NNR.The number of trainable LoRA parameters depends on rank and the convolution-layer shapes.Source: B. Content adaptation · p. 3
  • Per-sequence adaptation fine-tunes the update using decoded LR frames, per-slice QP maps, and original HR frames as supervision.The shared component-wise MSE loss uses weights wY = 4 and wUV = 2 for the luma and chroma residual streams.Source: B. Content adaptation · p. 3

C. NNR coding

CASR compresses each sequence’s adapted weight update with NNR and signals it as side information. NNR quantization is selected through an encoder-side rate–distortion search.

C. NNR coding

  • Per-sequence LoRA matrices or Multiplier vectors are encoded as weight updates using NNR, then signalled in the bitstream.Uniform-scalar quantization is followed by DeepCABAC entropy coding.Source: C. NNR coding · p. 3
  • An inner rate–distortion search refines each tensor’s NNR QP to limit training-loader reconstruction-loss change while reducing bitstream length.The selected QP is signalled in the payload.Source: C. NNR coding · p. 3
  • Table I compares BD-rate and relative encoding/decoding time against VTM, with LoRA outperforming Multiplier on chroma components.Source: Table I · p. 4

A. Experimental setup

The experiments pretrain configuration-specific base SR networks, then overfit only adaptation parameters for each test sequence. Evaluation uses JVET Class A1 and A2 sequences under random-access and all-intra settings.

A. Experimental setup

  • The base SR network is pretrained on DIV2K for AI and on UHD subsets of BVI-DVC and TVD for RA, for up to 200 epochs.Training uses Adam with learning rate 1e−3 and batch size 256.Source: A. Experimental setup · p. 4
  • For each test sequence, only LoRA matrices or multiplier scalars are fine-tuned while base-model weights remain frozen.Overfitting runs for up to 200 epochs on 64×64 LR patches with batch size 16; LoRA uses r=4 and α=4.Source: A. Experimental setup · p. 4
  • Evaluation covers six 3840×2160 JVET CTC Class A1/A2 sequences under RA and AI, using five QPs per configuration and native-resolution VTM as the anchor.Proposed methods downsample sequences by ×2; their bitrates include NNR side information.Source: A. Experimental setup · p. 4

B. Overall Results and Analysis

LoRA adaptation improves over non-adapted SR and provides BD-rate savings across luma and chroma components. The results also show chroma gains over Multiplier adaptation, alongside encoder-side fine-tuning costs and improved visual fidelity.

B. Overall Results and Analysis

  • RPR and non-adapted Base SR provide luma gains but degrade chroma, whereas overfitting mitigates the chroma deficiencies.The RPR baseline achieves luma savings of −3.94% for RA and −4.04% for AI.Source: B. Overall Results and Analysis · p. 4
  • −10.93% (Y) under RA and −13.43% (Y) under AI are the LoRA BD-rate results; under RA, LoRA also reaches −15.39% (U) and −24.41% (V).LoRA achieves consistent savings across all three components under both configurations.Source: B. Overall Results and Analysis · p. 4
  • LoRA improves on Multiplier particularly for chroma, where scalar gain adjustment has limited capacity to capture spatially varying colour patterns.Multiplier retains some positive AI chroma BD-rates, including +7.77% (U).Source: B. Overall Results and Analysis · p. 4
  • Adaptation raises encoding time to 145–176% of VTM, while LoRA decoding matches Base SR because its updates are merged into model parameters.Both adaptive methods retain the Base SR architecture, with most additional cost on the encoder side.Source: B. Overall Results and Analysis · p. 4
  • On ParkRunning3, the LoRA-adapted RD curve is above non-adapted SR across the operating range, with content-adaptive variants outperforming the SR base.The gap to native-resolution encoding narrows at high bitrates.Source: B. Overall Results and Analysis · pp. 4–5, Figure 4 · p. 4
  • At matched bitrate, LoRA better preserves fine texture and colour fidelity than Base SR or Multiplier in the ParkRunning3 crop.The rank study compares BD-rate, trainable parameters, and NNR signalling overhead under RA and AI.Source: Figure 5 · p. 5, B. Overall Results and Analysis · p. 5, Table II · p. 5

C. LoRA rank and signalling overhead

The section compares LoRA ranks and their signalling costs, selecting r=4 as the operating point. LoRA’s adaptation also progressively recovers chroma degradation, outperforming Multiplier across components.

C. LoRA rank and signalling overhead

  • Even at r=1, LoRA outperforms Multiplier; r=4 steadily improves BD-rate, while r=8 brings diminishing returns with a larger NNR payload.Source: C. LoRA rank and signalling overhead · p. 5
  • Approximately 41 kbps is the average NNR-coded payload for LoRA (r=4) across QPs, versus approximately 9 kbps for Multiplier.The absolute overhead is nearly QP-independent, so its relative percentage grows at higher QPs as the base SR bitstream becomes smaller.Source: C. LoRA rank and signalling overhead · p. 5
  • Figure 6 shows content adaptation progressively recovering chroma degradation from Base SR, with LoRA outperforming Multiplier across Y, U, and V.Source: Figure 6 · p. 5
  • r=4 is selected as the default, balancing rank-driven coding gains against NNR payload size.At r=8, gains diminish while payload size increases.Source: C. LoRA rank and signalling overhead · p. 5

D. Overfitting convergence

During overfitting, both methods improve all components, with most gains arriving early; LoRA consistently outperforms Multiplier, especially on chroma.

D. Overfitting convergence

  • Most gains materialise within the first 100 epochs, at approximately 110% encoding time for RA and 120% for AI.Figure 6 traces BD-rate evolution during overfitting; both methods improve all components, with LoRA most clearly outperforming Multiplier on chroma.Source: D. Overfitting convergence · p. 5
  • ReduceOnPlateau further refines the result by lowering the learning rate when validation loss stagnates.Source: D. Overfitting convergence · p. 5

IV. CONCLUSION

CASR is a VVC-compatible content-adaptive SR post-filter with internal ×2 upscaling and encoder-side per-sequence overfitting. On JVET CTC class A1 and A2 under RA and AI, LoRA improves over non-adapted SR and multiplier-only overfitting, with r=4 balancing coding gain and signalling overhead.

IV. CONCLUSION

  • LoRA-based overfitting improves over both a non-adapted SR post-filter and multiplier-only overfitting on JVET CTC class A1 and A2 sequences under RA and AI.Source: §IV · p. 5
  • CASR performs ×2 upscaling internally and is compliant with VSEI NNPF input/output tensors and signalling.Its base network uses sub-pixel convolution.Source: §IV · p. 5
  • Per-sequence encoder-side adaptation overfits LoRA matrices or Multipliers on selected convolution layers, then compresses the weight update with NNR for side-information signalling.Source: §IV · p. 5
  • Rank r=4 provides a favourable trade-off between coding gain and signalling overhead.Source: §IV · p. 5
Loading 2609.37328v1…