PaperlayerView exact revision on arXiv

Source-linked AI summary

Adaptive Fused Prior Transfer for Controllable Generative Image Compression

Yifei Pei, Ying Liu, Nam Ling

arXiv:2605.16817v4eess.IVcs.CV

TL;DR

At very low bitrates, compression can lose local detail, and controllable codebook-based generative codecs commonly rely on single-codebook priors. AFP-GIC transfers an adaptive fused prior from frozen AdaCode, guiding the encoder and predicting a compatible decoder prior without transmitting it. The paper reports improved no-reference naturalness alongside trade-offs in other quality metrics, and identifies scope limits tied to fixed training data and budget.

  • Problem

    At very low bitrates, compressed images lose fine textures and local structures, while existing controllable codebook-based designs generally rely on single-codebook reconstruction priors.

    Source: §I · pp. 1–2

  • Method

    AFP-GIC transfers a frozen AdaCode adaptive fused prior through encoder-side guidance and decoder-side prior prediction, without transmitting the prior itself.

    Source: §I · p. 2, §VI · p. 18

  • Results

    AFP-GIC achieves superior NIQE naturalness to DC-VIC, while DC-VIC retains better MS-SSIM, DISTS, and FID results.

    Source: §V · p. 18

  • Takeaways & Limitations

    The evidence supports adaptive fused-prior transfer as a practical direction for controllable low-bitrate perceptual compression.

    Source: §VI · p. 18

    Experiments use a fixed training dataset and budget, leaving scaling with larger datasets or longer schedules unexamined.

    Source: §V · p. 18

Abstract

from arXiv · show

At very low bitrates, image compression discards fine textures and local structures, while distortion-oriented reconstruction often produces over-smoothed images. Generative codecs synthesize missing details, but existing codebook-based controllable designs generally rely on single-codebook reconstruction priors. We propose Adaptive Fused Prior Transfer for Controllable Generative Image Compression (AFP-GIC), which transfers an image-adaptive fused prior from a frozen pretrained AdaCode model. Encoder-side prior features guide latent formation, while the decoder predicts a compatible fused prior from the compressed representation and control variables, without transmitting the prior itself. A motivating analysis shows that better decoder-side prior alignment tightens a reconstruction-error upper bound and that the fused-prior family includes single-codebook choices as special cases. A single pretrained model supports five evaluated bitrate operating points. Under the unified benchmark, AFP-GIC achieves 18.1% lower decoder latency and uses 31.10 million (20.5%) fewer inference parameters than DC-VIC. Experiments on Kodak, CLIC2020, and DIV2K show competitive PSNR and SSIM, with the clearest naturalness gains in NIQE scores and very-low-bitrate visual comparisons. Code: https://github.com/yifeipet/AFP_GIC.

I. INTRODUCTION

AFP-GIC replaces a fixed single-codebook reconstruction prior with adaptive fused-prior transfer, using encoder-side guidance and decoder-side prediction. The introduction motivates this design by the difficulty of preserving local detail at very low rates and by the need to make generative reconstruction controllable.

I. INTRODUCTION

  • Below roughly 0.2 bpp, compressed representations often lack fine textures and local structures, while distortion-oriented reconstruction can produce over-smoothed images.The perception-distortion trade-off means higher PSNR does not necessarily imply visually sharper reconstructions.Source: §I · pp. 1–2

B. PERCEPTUAL AND GENERATIVE IMAGE COMPRESSION

  • Perceptual and generative codecs use learned priors to synthesize missing detail, but fixed operating preferences, decoding cost, and fidelity trade-offs remain concerns.GAN-based training can introduce structured artifacts, while diffusion approaches may be computationally expensive or relax reference fidelity.Source: §I · p. 2, B. PERCEPTUAL AND GENERATIVE IMAGE… · p. 3

I. INTRODUCTION

  • Controllable generative codecs leave open how to represent reconstruction priors and make them available at the decoder under entropy-constrained decoding.Adaptive codebook fusion can represent a broader range of structures and textures than a single-codebook choice.Source: §I · p. 2
  • AFP-GIC transfers a frozen AdaCode adaptive fused prior, guiding encoder latent formation while predicting a compatible prior at the decoder without transmitting it.The formulation generalizes controllable prior-guided compression beyond single-codebook priors.Source: §I · p. 2
  • The paper motivates fused-prior alignment with a tighter reconstruction-error upper bound and pairs prior transfer with a lightweight decoder to reduce latency and parameters versus DC-VIC.The analysis assumes a Lipschitz decoder; the efficiency comparison is reported for the paper's evaluation.Source: §I · p. 2

A. LEARNED IMAGE COMPRESSION

  • Learned image codecs build on transform coding and entropy modeling, but rate-distortion optimization alone does not target perceptual reconstruction at very low rates.Pixel-fidelity objectives can favor averaged outputs when local structure is not encoded.Source: §I · p. 1, A. LEARNED IMAGE COMPRESSION · p. 3

C. CONTROLLABLE IMAGE COMPRESSION

Controllable compression uses one model across operating preferences, while pretrained codebook priors supply reconstruction guidance for missing detail. AFP-GIC transfers a frozen AdaCode fused prior through encoder-side guidance and decoder-side prediction, adapting that prior to the compressed representation without transmitting it.

C. CONTROLLABLE IMAGE COMPRESSION

  • A single controllable model can cover multiple bitrate and reconstruction preferences, but the choice of generative prior and its availability at the decoder remain central design issues.Prior work includes decoder-side realism control, joint rate and reconstruction-behavior control, and dual-conditioned compression.Source: C. CONTROLLABLE IMAGE COMPRESSION · p. 3

D. PRETRAINED GENERATIVE PRIORS AND CODEBOOK-BASED MODELS

  • Single-codebook priors select one entry per location, whereas AdaCode combines multiple codebooks with spatially varying weights to represent more varied structures and textures.The decoder-side information constraints on using adaptive fusion in compression remain less explored.Source: D. PRETRAINED GENERATIVE PRIORS AND… · p. 3

A. OVERALL FRAMEWORK

  • AFP-GIC combines an entropy-constrained compression backbone with a frozen AdaCode prior, deriving encoder guidance from the image and predicting a compatible prior at the decoder.The decoder lacks both the original image and the encoder-side fused prior, so the fused prior itself is not transmitted.Source: §III · p. 4, A. OVERALL FRAMEWORK · p. 4

B. ENCODER-SIDE ADAPTIVE FUSED-PRIOR TRANSFER

  • AdaCode quantizes image features through multiple codebooks and predicts spatial fusion weights whose weighted combination forms the image-adaptive fused prior.The fused representation is continuous and can cover a broader set of local structural and textural patterns than one codebook entry.Source: B. ENCODER-SIDE ADAPTIVE FUSED-PRIOR… · p. 5
  • The Prior Feature Adapter resizes and projects the fused prior into the compression encoder's feature space, where it guides latent formation.The adapter injects the adapted prior after the encoder's third downsampling layer.Source: B. ENCODER-SIDE ADAPTIVE FUSED-PRIOR… · p. 5

C. DECODER-SIDE PRIOR PREDICTION AND GUIDED RECONSTRUCTION

Because the original image and its fused prior are unavailable at decoding, AFP-GIC predicts a compatible prior from the quantized latent and uses it with separate control-dependent modulation features for reconstruction.

C. DECODER-SIDE PRIOR PREDICTION AND GUIDED RECONSTRUCTION

  • The decoder estimates a compatible fused prior from the quantized latent because it cannot access the original image or ground-truth prior.The predicted representation is in the AdaCode-compatible feature space.Source: C. DECODER-SIDE PRIOR PREDICTION AND… · pp. 5–6
  • An SFT extractor derives modulation features from the quantized latent and both controls, while the predicted prior is supplied to the frozen AdaCode decoder.The SFT features generate spatially varying scale and shift parameters for intermediate decoder features.Source: C. DECODER-SIDE PRIOR PREDICTION AND… · p. 6
  • The fused prior is not transmitted, so the entropy-coded bitstream uses compressed latents and compact header information while retaining prior-guided reconstruction.Source: C. DECODER-SIDE PRIOR PREDICTION AND… · p. 6

D. DESIGN CONSIDERATIONS OF PRIOR TRANSFER

AFP-GIC uses asymmetric prior-transfer modules to address the different information available at encoding and decoding, with both control variables influencing processing on both sides.

D. DESIGN CONSIDERATIONS OF PRIOR TRANSFER

  • At the encoder, a lightweight adapter resizes the image-derived fused prior to the feature grid and projects it into encoder channels.This targets scale and channel mismatch while keeping the encoder-side transfer module small.Source: D. DESIGN CONSIDERATIONS OF PRIOR… · p. 6
  • At the decoder, a dedicated estimator infers a compatible prior from compressed features and controls, refining features with residual blocks and a coarse-information skip branch.Source: D. DESIGN CONSIDERATIONS OF PRIOR… · p. 6
  • Prior consistency aligns encoder-side and predicted priors for the frozen decoder; continuous fused-feature regression avoids hard single-codebook selection and supports a lightweight fully convolutional decoder.The paper reports this decoder design reduces parameters and decoding latency relative to DC-VIC’s Transformer-based decoder.Source: D. DESIGN CONSIDERATIONS OF PRIOR… · p. 6

E. DUAL-CONTROL FORMULATION

  • The bitrate and prior controls are Fourier-embedded, combined by an MLP, and jointly condition both encoder and decoder processing.They affect latent formation, decoder-side prior prediction, and SFT modulation rather than being assigned separate encoder-only and decoder-only roles.Source: E. DUAL-CONTROL FORMULATION · pp. 6–7

F. QUANTIZATION AND ENTROPY CODING

AFP-GIC retains a hyperprior-based quantization and entropy-coding pipeline, then trains with rate, distortion, perceptual, prior-consistency, and adversarial objectives.

F. QUANTIZATION AND ENTROPY CODING

  • The codec rounds and arithmetic-codes quantized hyper-latents and main latents, modeling the former with an entropy bottleneck and the latter slice-wise with conditional Gaussian parameters.Main-latent parameters use decoded hyperprior features and previously reconstructed slices.Source: F. QUANTIZATION AND ENTROPY CODING · p. 7
  • The bitrate term sums negative log-likelihoods for the quantized hyper-latent and main latent; the selected control pair is conveyed in a compact header excluded from the training-time rate term.The adaptive fused-prior mechanism is built on this pipeline rather than replacing it.Source: F. QUANTIZATION AND ENTROPY CODING · p. 7

G. OBJECTIVE FUNCTIONS AND DISCRIMINATOR

  • Non-adversarial training combines rate, pixel-domain MSE distortion, AlexNet-based LPIPS, and a prior-consistency loss, with controls adjusting rate and prior-alignment emphasis.The rate and prior weights remain positive and vary smoothly and monotonically with their respective controls.Source: G. OBJECTIVE FUNCTIONS AND DISCRIMINATOR · p. 7
  • Adversarial refinement uses a control-conditioned PatchGAN that evaluates local regions, and begins only after initial non-adversarial training.During refinement, the encoder and entropy models are fixed and the rate term is omitted; the staged ordering is intended to reduce destabilization risk.Source: G. OBJECTIVE FUNCTIONS AND DISCRIMINATOR · pp. 7–8

H. TRAINING STRATEGY

AFP-GIC trains across sampled control pairs, selects bitrate-specific pairs on validation data, and fine-tunes on that compact set for evaluation at five bitrate points.

H. TRAINING STRATEGY

  • AFP-GIC uses sampled-pair training, validation-based operating-point selection, and selected-pair fine-tuning rather than optimizing only one fixed operating point.The schedule is coupled with adaptive fused-prior transfer and decoder-side prior alignment.Source: H. TRAINING STRATEGY · p. 8

Stage I: DUAL-CONDITIONED TRAINING WITH SAMPLED CONTROL PAIRS

  • Stage I samples bitrate and prior controls across discrete grids, with 1.0M non-adversarial iterations followed by 500K adversarial-refinement iterations.Source: H. TRAINING STRATEGY · p. 8

Stage II: VALIDATION-BASED BETA SELECTION

  • Stage II selects one control pair per target bitrate by matching validation bitrate, then ranking candidates with a PSNR–FID score.FID is used only for validation-time pair selection, not as a primary reported metric, because it is set-level and sensitive to sample size and protocol.Source: H. TRAINING STRATEGY · p. 8

Stage III: SELECTED-PAIR FINE-TUNING

  • Stage III fine-tunes only on the selected control-pair set with adversarial and prior-consistency supervision, omitting the rate term, and is evaluated as a five-point variable-rate codec.Source: H. TRAINING STRATEGY · p. 8

I. MOTIVATING ANALYSIS

The analysis motivates adaptive fused-prior transfer through a decoder-side prior-alignment bound and the broader set of priors available through adaptive fusion.

I. MOTIVATING ANALYSIS

  • Adaptive fusion contains every single-branch choice and is strictly richer when branch features are non-degenerate.The single-codebook case is recovered with one-hot fusion weights.Source: §I · p. 9
  • Under a Lipschitz decoder assumption, the reconstruction-error bound separates prior mismatch from the residual under the ideal prior.A predicted prior closer to the ideal prior tightens the bound.Source: §I · p. 9
  • The input-image fused prior supervises the predicted prior, but the analysis does not assume that this target equals the ideal prior.Source: §I · p. 9

IV. EXPERIMENTS

Across Kodak, CLIC2020, and DIV2K, AFP-GIC shows its clearest perceptual advantages in NIQE and very-low-bitrate visual comparisons, while fidelity comparisons are mixed. The experiments evaluate five operating points and report decoder-side training stability at approximately 0.05 bpp.

A. EXPERIMENTAL SETUP

B. QUANTITATIVE COMPARISON WITH REPRESENTATIVE CODECS

A. EXPERIMENTAL SETUP

  • At approximately 0.05 bpp, final in-loop validation varied by 0.009 dB in PSNR across 11 checkpoints, indicating a stable plateau but not independent-seed variability.Source: A. EXPERIMENTAL SETUP · p. 11

B. QUANTITATIVE COMPARISON WITH REPRESENTATIVE CODECS

  • AFP-GIC reports negative BD-NIQE versus MS-ILLM on all three datasets, while comparisons with DC-VIC and CRDR are mixed.Against DC-VIC, it has higher single-scale BD-SSIM but slightly worse BD-DISTS, while BD-MS-SSIM and FID favor DC-VIC; CRDR leads BD-PSNR, while AFP-GIC remains competitive in BD-LPIPS.Source: Table 4 · p. 11, B. QUANTITATIVE COMPARISON WITH… · p. 11

C. QUALITATIVE COMPARISON

Across the qualitative comparisons, AFP-GIC preserves local structure at very low bitrates. The complexity and ablation analyses report lower decoding cost than DC-VIC and examine the effects of prior alignment, control settings, and model components.

C. QUALITATIVE COMPARISON

D. COMPLEXITY AND RUNTIME ANALYSIS

E. ABLATION STUDY

  • Removing inherited SFT modulation causes the largest full-reference quality losses, while adversarial supervision yields cleaner local structures and more coherent textures in a representative example.Source: E. ABLATION STUDY · pp. 14, 18, Figure 11 · p. 18
  • Across Kodak, CLIC2020, and DIV2K, AFP-GIC improves PSNR, SSIM, LPIPS, and DISTS over single-prior C2, while C2 yields slightly lower NIQE.The five-point BD comparison uses each dataset’s common bitrate interval.Source: Table 8 · p. 15, E. ABLATION STUDY · p. 15
  • At fixed βrate, increasing βprior generally improves PSNR only slightly; prior-consistency loss preserves responsiveness, but effects are ≤0.1 dB and are not rate-matched RD gains.Source: E. ABLATION STUDY · pp. 14–16
  • Reducing predicted-prior mismatch raises mean PSNR from 26.994 dB to 27.363 dB, with positive endpoint gains across all 120 image–operating-point combinations.The ground-truth fused prior is unavailable during practical decoding; the test changes no bitstream or other decoder inputs.Source: E. ABLATION STUDY · p. 17

V. DISCUSSION AND FUTURE WORK

At very low bitrates, AFP-GIC improves no-reference naturalness while retaining trade-offs in other quality metrics, and it supports flexible rate control with efficient repeated decoding. Its limits include fine-detail errors at the lowest rate, dependence on the frozen AdaCode prior, and untested settings beyond its severe-information-bottleneck focus.

V. DISCUSSION AND FUTURE WORK

  • AFP-GIC yields superior NIQE naturalness, while DC-VIC remains better on MS-SSIM, DISTS, and FID, reflecting a trade-off across complementary criteria.NIQE differences between released and locally trained DC-VIC also show sensitivity to training configuration.Source: §V · p. 18
  • At 0.066 bpp on Kodak, AFP-GIC preserves the global scene but distorts text and fine structures; such errors become less visible at higher rates.Symbol-level and fine geometric details remain difficult when little direct information is available.Source: §V · p. 18, Figure 12 · p. 18
  • One trained AFP-GIC model supports all reported operating points, adapting to bitrate needs without separate bitrate-specific models and serving repeated-decoding use cases.The discussion frames its practical value as extending beyond low-rate naturalness to flexible rate control and efficient repeated decoding.Source: §V · p. 18
  • Experiments use a fixed training dataset and budget, leaving scaling with larger datasets or longer schedules unexamined; behavior also depends on the frozen AdaCode prior.Replacing the predicted prior with the ground-truth fused prior yields an approximately 0.39 dB BD-PSNR gain; loss terms remain coupled, and subjective preference studies are outside scope.Source: §V · p. 18
  • AFP-GIC targets severe information bottlenecks, so its behavior at higher bitrates remains an open question.The paper proposes exploring the transition as details can be transmitted more directly, alongside stronger external priors, better prior estimation, and more data-efficient control selection.Source: §V · p. 18

VI. CONCLUSION

AFP-GIC transfers a frozen AdaCode adaptive fused prior through encoder-side guidance and decoder-side prediction, without transmitting the prior. It offers lower decoder latency than DC-VIC, at the cost of higher encoder latency, supporting its use for controllable low-bitrate perceptual compression.

VI. CONCLUSION

  • AFP-GIC combines encoder-side fused-prior guidance with decoder-side prior prediction, handling prior asymmetry without transmitting the fused prior.Under the 256 × 256 benchmark, decoder latency is lower than DC-VIC, while encoder latency is higher (81.3 vs. 61.6 ms).Source: §VI · p. 18

APPENDIX A NUMERICAL VALUES UNDERLYING THE QUANTITATIVE COMPARISON

The appendices provide operating-point values and supplementary evaluations, including bitrate-normalized quality–complexity comparisons and prior-substitution diagnostics. They also compare AFP-GIC with DC-VIC under a matched local training procedure and report supplementary FID evaluations.

APPENDIX A NUMERICAL VALUES UNDERLYING THE QUANTITATIVE COMPARISON

  • Tables 12–14 provide numerical values for evaluated operating points underlying Fig. 3 on Kodak, CLIC2020, and DIV2K.The values use available evaluated points within or near AFP-GIC’s bitrate range, without synthetic points.Source: Appendix A · p. 18, Table 12 · p. 19, Table 13 · p. 19

APPENDIX B SUPPLEMENTARY FID EVALUATION

  • FID is reported on CLIC2020 and DIV2K, not Kodak; supplementary comparisons include directly evaluated low-bitrate points and reproduced Control-GIC points.FID uses 2048-dimensional Inception-v3 features and patch grids; Tables 15–16 give dataset values, while Fig. 13 compares learned methods.Source: Appendix B · p. 19, Figure 13 · p. 19, Table 15 · p. 20

APPENDIX C PARTIAL COMPARISON WITH CONTROL-GIC

  • The partial Control-GIC comparison uses two released-checkpoint granularity settings and reports actual bpp, without computing BD metrics.Table 17 contains the partial comparison.Source: Appendix C · p. 20, Table 17 · p. 20

APPENDIX D BITRATE-NORMALIZED QUALITY–COMPLEXITY TRADE-OFF

  • AFP-GIC has 120.6M parameters, with mean BD-PSNR of +0.120 dB and mean BD-NIQE of -0.176; CRDR has higher mean BD-PSNR, while MS-ILLM decodes fastest.The bitrate-normalized comparison averages across Kodak, CLIC2020, and DIV2K; quality and complexity dimensions do not yield a scalar ranking.Source: Appendix D · p. 20, Figure 14 · p. 20

DIAGNOSTIC

  • Zero and spatial-mean prior substitutions degrade all five metrics, while the ground-truth fused-prior oracle improves four full-reference metrics but slightly raises NIQE.The substitutions hold the bitstream, controls, SFT features, decoder, and model parameters fixed; the oracle is non-deployable.Source: DIAGNOSTIC · p. 20, Table 18 · p. 20

APPENDIX F SUPPLEMENTARY COMPARISON UNDER THE DC-VIC TRAINING PIPELINE

  • With the same local training procedure and data, AFP-GIC outperforms the DC-VIC reproduction across most operating bitrates in PSNR, SSIM, LPIPS, and NIQE.Against the released checkpoint, AFP-GIC shows favorable PSNR, SSIM, and NIQE trends, while that checkpoint remains stronger in DISTS and FID; AFP-GIC uses frozen AdaCode pretraining.Source: Appendix F · pp. 20–21, Figure 15 · p. 21

I. SNR AND MS-SSIM RESULTS

The supplementary section reports SNR and MS-SSIM evaluations, explains the evaluation metrics and header accounting, and assesses image-sampling uncertainty in Kodak comparisons. In that uncertainty analysis, the BD-PSNR gain over DC-VIC is significant, while the favorable BD-NIQE estimate is not significant at 95% confidence.

I. SNR AND MS-SSIM RESULTS

  • The SNR and MS-SSIM supplement provides rate–quality curves, BD values relative to MS-ILLM, and pointwise values for Kodak, CLIC2020, and DIV2K.The BD-SNR calculation follows the common-bitrate protocol, and BD-SNR equals BD-PSNR up to numerical precision because the same images are evaluated.Source: §I · p. 24, Figure S1 · p. 24, Table S1 · p. 25

II. TAXONOMY OF EVALUATION METRICS

  • The evaluation taxonomy spans full-reference pixel and structural fidelity, full-reference perceptual similarity, no-reference naturalness, and distributional realism.The section cautions that a single metric is insufficient in the very-low-bitrate regime because of the rate-distortion-perception trade-off.Source: §II · p. 26

III. HEADER-OVERHEAD ACCOUNTING

  • AFP-GIC’s fixed 48-bit header encodes image dimensions, the maximum absolute quantized-latent value, and an operating-point index; dataset header fractions are averaged across images.The per-image fraction is defined relative to total realized bitstream length.Source: §III · p. 26

IV. PAIRED UNCERTAINTY ANALYSIS ON KODAK

  • AFP-GIC’s BD-PSNR gain over DC-VIC is +0.233 dB (95% CI: [+0.176, +0.304] dB), while its BD-NIQE difference is −0.069 (95% CI: [−0.154, +0.013]).The BD-PSNR gain is statistically significant; the favorable BD-NIQE estimate does not reach 95% confidence. Intervals quantify image-sampling uncertainty, not training-seed variability, and are unadjusted for multiple comparisons.Source: §IV · p. 26, Table S6 · p. 27
Loading 2605.16817v4…