PaperlayerView exact revision on arXiv

Source-linked AI summary

Gen2-VC: Unlocking Generative Priors for Video Compression

Yinhuan Huang, Jingkai Ying, Pu Chen, Zhijin Qin

arXiv:2609.34725v1eess.IV

TL;DR

At stringent bitrates, existing codecs struggle to balance source fidelity and perceptual realism, and generative designs can be codec-specific. Gen2-VC refines native codec reconstructions with a pretrained video prior; on six-dataset averages, it improves perceptual metrics and surpasses VTM-23.0 in PSNR and MS-SSIM.

  • Problem

    At stringent bitrates, distortion-oriented codecs can oversmooth details, while generative codecs risk source deviations and often use codec-specific designs.

    Source: §1 · p. 2

  • Method

    Gen2-VC uses native pixel-space codec reconstructions to anchor a pretrained video prior, with lightweight spatial and temporal LoRA adaptation while preserving codec bitstreams and reference updates.

    Source: §1 · p. 3, §3.1 · pp. 4–5

  • Results

    86.65%/94.24% bitrate savings at matched LPIPS/DISTS versus VTM-23.0; on six-dataset averages, Gen2-VC-DCVC-UF also surpasses it in PSNR/MS-SSIM.

    Source: §4.2 · p. 7

  • Takeaways & Limitations

    Adapters trained only on DCVC-UF deliver perceptual gains on DCVC-RT, ECM, VTM, and HM without modifying their bitstreams or reference update chains.

    Source: §1 · p. 3, §4.2 · p. 7

Abstract

from arXiv · show

Under stringent bitrate constraints, existing video codecs struggle to balance source fidelity and perceptual realism. Distortion-oriented codecs often oversmooth details, while generative codecs risk introducing content and structural deviations and rely on codec-specific designs. This motivates a question: Can existing codecs achieve a better distortion--perception trade-off through simple, reusable adaptation? Our insight is that native codec reconstructions provide a shared interface through which generative refinement is anchored to source content while remaining decoupled from codec-specific representations. We therefore propose Gen2-VC, a generative video compression framework that enhances the outputs of learned and conventional codecs with a pretrained video prior, leaving their bitstreams and reference update processes unchanged. With the codec, VAE, and generative backbone frozen, lightweight LoRA adapters refine codec reconstruction through single-frame spatial adaptation followed by multi-frame temporal adaptation with the video prior. Using Wan2.1-T2V-1.3B, Gen2-VC-DCVC-UF outperforms previous leading codecs in LPIPS/DISTS and, to our knowledge, is the first generative video codec to surpass VTM-23.0 in both PSNR and MS-SSIM at low bitrates, based on BD-rates averaged over six datasets. Compared to VTM-23.0, it reduces bitrate by an average of 86.65% and 94.24% at matched LPIPS and DISTS, respectively. Adapters trained only on DCVC-UF improve perceptual quality on DCVC-RT, ECM, VTM, and HM without retraining.

1 INTRODUCTION

At low bitrates, distortion-focused codecs can oversmooth details while generative codecs risk source deviations and codec-specific designs. Gen2-VC adapts a pretrained video prior through native reconstructions, improving the distortion–perception trade-off and transferring across codecs.

1 INTRODUCTION

  • Distortion-oriented codecs can oversmooth textures and details, while generative codecs may introduce content or structural deviations and rely on codec-specific designs.These trade-offs motivate the question of whether existing codecs can improve distortion–perception balance through reusable adaptation.Source: §1 · p. 2
  • Native codec reconstructions provide a shared pixel-space interface that anchors generative refinement to source content while decoupling it from codec-specific representations and reference updates.The reconstructions retain basic source content and spatial layout, supporting lightweight adaptation and reuse.Source: §1 · p. 2
  • Gen2-VC preserves the codec’s native reference update chain and bitstream while adapting a pretrained video prior with the codec, VAE, and backbone frozen.Lightweight LoRA adapters learn spatial refinement first, followed by temporal refinement; the I-adapter remains fixed during the second stage.Source: §1 · pp. 2–3
  • Gen2-VC-DCVC-UF improves the low-bitrate distortion–perception trade-off across six datasets and transfers DCVC-UF-trained adapters to other learned and conventional codecs.The paper reports leading LPIPS/DISTS performance alongside PSNR and MS-SSIM gains over VTM-23.0.Source: §1 · p. 3

2 RELATED WORK

Prior work improves video compression or restoration through conventional and learned codecs, generative codecs, and generative restoration. Gen2-VC instead refines native pixel-space reconstructions, using a shared interface across codecs.

Distortion-oriented video compression

  • Conventional and learned distortion-oriented codecs use varied coding approaches, while Gen2-VC applies generative refinement to their native reconstructions without changing their coding pipelines.The described codec families include HM, VTM, ECM, DVC, FVC, DCVC, DCVC-RT, and DCVC-UF.Source: §2 · p. 3

Generative video compression

  • Generative codecs use perceptual or adversarial training, or pretrained diffusion and flow models, but their generation is tied to codec-specific representations and reconstruction pipelines.Gen2-VC instead applies refinement after independent codec decoding through native pixel-space reconstructions.Source: §2 · p. 3

Generative video restoration

  • Generative video restoration targets detail recovery and super-resolution, but generic restoration training may not match codec distortions or low-bitrate fidelity–perception needs.Gen2-VC focuses on lightweight adaptation of a shared frozen video prior and training-free transfer across learned and conventional codecs.Source: §2 · p. 3

Generative video compression

  • Figure 3 depicts the I-adapter restoring individual frames before the P-adapter jointly refines them, alongside sliding-window recurrence and output selection.The recurrence and selection are shown with zero-based frame indices.Source: Figure 3 · p. 4

3 METHODS

Gen2-VC refines native codec reconstructions with shared frozen VAE and video-prior components while leaving the codec’s bitstream and reference updates unchanged. Its I-adapter performs spatial refinement before the P-adapter performs recurrent temporal refinement, trained in separate stages.

3 METHODS

  • Gen2-VC decodes a source video to native reconstructions, refines frames independently with the I-adapter, then jointly refines successive frames with the P-adapter.The P-adapter reuses its outputs as recurrent context while preserving the codec’s bitstream and reference-update process.Source: §3 · p. 4

3.1 SOURCE-ANCHORED SPATIAL ADAPTATION

  • The I-adapter independently encodes each native reconstruction, applies one near-terminal latent update without added noise, and decodes the result into an enhanced frame.The update uses a fixed conditioning shared across videos, with fixed refinement strength for each operating point.Source: §3.1 · pp. 4–5

3.2 RECURRENT TEMPORAL ADAPTATION

  • The P-adapter refines overlapping frame windows using prior gated outputs as history, then finalizes Lin new enhanced frames per call in steady state.Each call combines history with new I-enhanced frames, applies a latent update, and gates the candidate window with its input.Source: §3.2 · p. 5

3.3 STAGED OPTIMIZATION

  • Training first learns spatial refinement with the I-adapter, then freezes it while the P-adapter learns temporal refinement; the codec, VAE, and backbone remain frozen.The shared objective combines source-fidelity, perceptual, and adversarial terms, with different adversarial-loss settings for the two adapters.Source: §3.3 · pp. 5–6

4 EXPERIMENTS

Across six datasets, Gen2-VC-DCVC-UF improves the low-bitrate distortion–perception trade-off and transfers perceptual gains across codec families. Experiments assess this approach against codec and restoration baselines, then ablate refinement stages and adapter ranks.

4.1 EXPERIMENTAL PROTOCOL

  • Testing covers six datasets, four full-reference quality metrics, distortion-oriented and generative codecs, and restoration models; adapters are trained on DCVC-UF for transfer evaluation.The protocol evaluates the first 96 frames of each sequence and computes BD-rate over the common quality range without extrapolation.Source: §4.1 · pp. 6–7

4.2 EXPERIMENTAL RESULTS

  • 86.65% and 94.24% bitrate savings at matched LPIPS and DISTS, respectively, and 12.11% and 28.03% at matched PSNR and MS-SSIM, versus VTM-23.0 on six-dataset averages.Gen2-VC-DCVC-UF also combines leading LPIPS/DISTS performance with PSNR/MS-SSIM gains over VTM-23.0 among evaluated generative codecs.Source: Table 1 · p. 7, §4.2 · p. 7
  • Gen2-VC outperforms SeedVR2, DiQP, and VEnhancer-v2 in available full-reference comparisons, while qualitative examples and a user study support subjective-quality gains.DCVC-UF can oversmooth details, whereas GVC1D and YODA can distort source content despite perceptual gains.Source: §4.2 · pp. 7–8
  • The full pipeline outpaces VoRTeC+ and restoration methods on one GPU, while refinement remains outside codec reference updates and adds decoder-side computation.The reported complexity comparisons distinguish enhancement-only costs and profile the I- and P-adapters separately.Source: §4.2 · p. 8, Table 2 · p. 8

4.3 ABLATION STUDY

  • With Second outputs, adding the P-adapter improves all four BD-rates over I-only and First, including LPIPS/DISTS changes from −88.35%/−96.52% to −92.06%/−97.47%.Second uses the second P-adapter estimate when available and otherwise the first across overlapping windows.Source: §4.3 · p. 9, Table 3 · p. 9
  • With Second outputs, staged I+P adaptation improves LPIPS/DISTS BD-rates from pixel-P’s −81.04%/−89.55% to −92.06%/−97.47%.Multi-frame training uses lower resolutions under fixed GPU memory; the fixed I-adapter preserves higher-resolution spatial refinement for the P-adapter.Source: §4.3 · p. 9, Table 3 · p. 9
  • The selected I-adapter rank is 16; under the same GPU memory budget, P-adapter rank 16 offers the best PSNR/MS-SSIM and comparable LPIPS/DISTS with 11.80M total LoRA parameters.P-adapter ranks 32 and 64 use 17.69M and 29.49M parameters, respectively; I-adapter rank 32 offers further gains but doubles its LoRA parameters.Source: §4.3 · p. 9, Table 3 · p. 9

5 CONCLUSION

Gen2-VC connects a pretrained video prior to existing codecs through native pixel-space reconstructions, preserving their bitstreams and reference update chains. Experiments show low-bitrate trade-off gains and training-free perceptual transfer across codec families.

5 CONCLUSION

  • Gen2-VC uses native pixel-space reconstructions to anchor a pretrained video prior’s refinement while preserving codec bitstreams and native reference update chains.The codec, VAE, and backbone are frozen; lightweight LoRA adapters perform spatial and temporal refinement in stages, with one latent update per call.Source: §5 · pp. 9–10
  • Across six datasets, one shared Wan2.1-T2V-1.3B prior improves low-bitrate rate–distortion–perception trade-offs, and DCVC-UF-trained adapters provide perceptual gains on other codecs without retraining.The paper identifies DCVC-RT, ECM, VTM, and HM as transfer targets.Source: §5 · p. 10
  • Gen2-VC provides a practical route to generative refinement in existing video compression systems without redesigning their coding pipelines.Source: §5 · p. 10

A APPENDIX

The appendix is organized as a reference for implementation, evaluation, ablations, user-study results, profiling, and visual comparisons.

A APPENDIX

  • Appendix A.1 covers implementation, training, and recurrent inference; A.2 covers codec configurations, color conversion, and metric implementations.Source: A APPENDIX · p. 15
  • Appendices A.2.2 and A.2.3 provide additional quality metrics and extended codec and restoration comparisons.Source: A APPENDIX · p. 15
  • Appendices A.3–A.7 cover ablations, user-study results, profiling, per-dataset BD-rates, and visual comparisons.Source: A APPENDIX · p. 15

A.1 IMPLEMENTATION AND TRAINING DETAILS

Implementation uses native RGB codec reconstructions as the refinement interface, with a frozen VAE and backbone and staged LoRA adaptation. Fixed controls and codec-quantization-based adapter selection govern inference.

A.1 IMPLEMENTATION AND TRAINING DETAILS

  • Gen2-VC takes native RGB codec reconstructions as its pixel-space interface and uses a frozen VAE to encode and decode refinement latents.Source: §A.1 · p. 15
  • The codec, VAE, backbone, and I-adapter during P-stage training remain frozen; LoRA targets self-attention Q/K/V/O projections in all 30 backbone blocks.The default training settings and adapter sizes are summarized in Table 4.Source: §A.1 · p. 15, Table 4 · p. 15
  • Refinement strengths and RGB gates are empirically set before training and held fixed; inference selects among four adapter pairs using the target codec’s quantization parameter.The controls transfer unchanged without codec-, dataset-, video-, or frame-specific tuning.Source: §A.1 · pp. 15–16
  • Each adapter call scales RGB inputs, encodes them with the VAE, performs one latent update without added noise, then decodes and rescales the output.The inference setup uses fixed conditioning and a specified near-terminal noise level.Source: §A.1 · p. 16

A.2 TESTING PROTOCOL AND METRIC IMPLEMENTATIONS

The protocol specifies low-delay conventional-codec configurations, encodes the first 96 frames of each sequence, and evaluates converted native reconstructions. It also documents baseline rate sources and a SeedVR2 color-correction setting.

Test command and evaluation

  • The evaluation encodes the first 96 frames of each 8-bit YUV 4:2:0 sequence at source resolution and converts source and native reconstruction frames to RGB8.Bitrate is computed from output bitstream size per frame area; framewise metrics are averaged over frames and then equally across sequences within each dataset.Source: §A.2 · p. 16, Table 6 · p. 17

Baseline availability and rates

  • HM, VTM, ECM, DCVC-RT, DCVC-UF, YODA, and GVC-RT use actual bitstream rates, while several generative baselines use entropy-model estimates.Gen2-VC and restoration methods retain their input codec’s bitrate; GLVC and GNVC-VD are excluded because code was unavailable at evaluation time.Source: §A.2 · p. 17

SeedVR2 color correction

  • SeedVR2 wavelet color correction improves six-dataset mean BD-rates for MS-SSIM, LPIPS, and DISTS on both codecs at identical bitrates.PSNR is omitted because its quality ranges do not overlap VTM-23.0; BD-rate uses quality sorting before PCHIP integration, including for non-monotonic curves.Source: §A.2 · p. 17, Table 7 · p. 17

A.2.1 METRIC IMPLEMENTATIONS

The section records metric implementations and explains how perceptual distribution metrics are aggregated across datasets and rates.

Metric implementations

  • FID and KID pool patches from all frames and sequences at each dataset/rate point rather than averaging framewise scores.Patch size is 64 × 64 when the dataset’s minimum frame dimension is below 256, and 256 × 256 otherwise.Source: §A.2.1 · p. 17
  • Table 8 records the metric libraries and explicit call settings used in the evaluation code.NIQE retries numerical linear-algebra failures on CPU, then averages finite frames and equally weights sequences with a defined mean.Source: §A.2.1 · p. 17, Table 8 · p. 18

A.2.2 ADDITIONAL QUALITY METRICS

Additional quality metrics indicate perceptual gains for Gen2-VC relative to native codec reconstructions across the evaluated datasets.

Additional quality metrics

  • Gen2-VC substantially reduces FID and KID across all six datasets, with lower NIQE and generally higher MUSIQ and CLIP-IQA at low bitrates.Figure 6 compares these metrics with distortion-oriented and generative codecs.Source: §A.2.2 · p. 18, Figure 6 · p. 19

A.2.3 EXTENDED CODEC AND RESTORATION COMPARISONS

The extended comparisons report cross-codec perceptual gains, stage and rank ablations, user-study preferences, computational costs, and per-dataset results. Qualitative comparisons also examine detail recovery, source structure, and restoration behavior across codecs and methods.

A.2.3 EXTENDED CODEC AND RESTORATION COMPARISONS

  • Adapters trained only on DCVC-UF improve LPIPS/DISTS on DCVC-RT, ECM, VTM, and HM without retraining, while Gen2-VC has a better fidelity–perception balance than restoration baselines given the same codec inputs.FID/KID are substantially lower than the restoration baselines, though SeedVR2 and VEnhancer-v2 attain higher MUSIQ or CLIP-IQA at some operating points.Source: §A.2.3 · p. 19, Figure 7 · p. 19, Figure 8 · p. 20

A.3.1 STAGES AND OUTPUT SELECTION

  • I+P Second improves all four BD-rates over I-only and I+P First on HEVC-E and in six-dataset averages, and achieves the lowest FloLPIPS BD-rate on all five reported datasets.These results support reconstruction and motion-sensitive perceptual gains from the P-stage.Source: §A.3.1 · p. 21, Table 9 · p. 21, Table 10 · p. 21
  • With equal total LoRA parameters, I+P improves mean LPIPS/DISTS BD-rates from pixel-P’s −81.97%/−90.49% to −86.65%/−94.24%.Pixel-P uses lower-resolution multi-frame training under fixed GPU memory and can lose perceptual quality at higher test resolutions; I+P preserves the independently trained spatial adapter’s refinement.Source: §A.3.1 · p. 21

A.3.2 LORA RANK

  • Rank 16 achieves the lowest mean BD-rate on each of four metrics with 11.80M total LoRA parameters, versus 17.69M and 29.49M for ranks 32 and 64.Under the same hardware, datasets, and training recipe, larger ranks offer no further mean BD-rate gains; rank 32 improves both perceptual metrics over rank 16 but doubles its LoRA parameters.Source: Table 9 · p. 21, §A.3.2 · p. 22

A.4 USER STUDY

  • Gen2-VC-DCVC-UF achieved the highest preference index, 90.00, among eight tested methods, at a joint-lowest mean rate of 0.00478 BPP.The index is model-based and does not represent a 90% observed win rate; the anonymous reference-guided study involved 30 adults and 1,920 formal judgments.Source: §A.4 · pp. 23–24, Table 11 · p. 23

A.5 COMPUTATIONAL COMPLEXITY

  • Complete single-GPU throughput exceeds profiled VoRTeC+ decoding and all three restoration stages, but trails S2VC/YODA at 480p–1080p.The reported costs cover enhancement-only stages on one A100 80 GB GPU, while the complete pipeline includes codec decoding followed by enhancement; two GPUs improve throughput without reducing MACs or window buffering.Source: Table 14 · p. 26, §A.5 · p. 27, Table 15 · p. 27
  • Frames retained for a Second estimate wait for the next overlapping call and four new inputs, adding buffering beyond native low-delay decoding.Reported FPS measures throughput, not per-frame output latency.Source: §A.5 · p. 27

A.6 PER-DATASET BD-RATES

  • Gen2-VC-DCVC-UF combines strong LPIPS/DISTS performance with substantially better PSNR/MS-SSIM than evaluated generative codecs across all six datasets.Training-free transfer also improves LPIPS and DISTS BD-rates over native DCVC-RT, ECM, VTM, and HM on all six datasets.Source: §A.6 · p. 27, Table 16 · p. 28, Table 17 · p. 29
Loading 2609.34725v1…