PaperlayerView exact revision on arXiv

Source-linked AI summary

Generative Refinement Networks for Visual Synthesis

Jian Han, Jinlai Liu, Jiahuan Wang, Bingyue Peng, Zehuan Yuan

arXiv:2604.13030v2cs.CV

TL;DR

Visual generation needs computation that adapts to content complexity while avoiding the tokenization losses and error accumulation of autoregressive models. GRN combines near-lossless HBQ with global refinement and entropy-guided adaptive-step generation, achieving strong ImageNet results and superior equivalent-scale text-to-image and text-to-video performance. The paper also identifies limitations in scale and some text-to-video detail and distortion.

  • Problem

    Diffusion models use uniform computation across samples, while autoregressive models are limited by lossy discrete tokenization and error accumulation.

  • Method

    GRN uses Hierarchical Binary Quantization for near-lossless discrete representations, global refinement for error correction, and entropy-guided sampling for complexity-aware adaptive-step generation.

  • Results

    GRN achieves state-of-the-art ImageNet reconstruction and class-conditional generation, with 0.56 rFID and 1.81 gFID, and surpasses equivalent-scale methods on text-to-image and text-to-video tasks.

  • Takeaways & Limitations

    GRN provides a visual synthesis framework combining discrete tokenization with refinement-based, complexity-aware generation across image and video tasks.

  • Takeaways & Limitations

    Training compute and model size were not scaled to leading visual-generation levels, and text-to-video outputs may lack rich detail or exhibit distortions in some cases.

Abstract

from arXiv · show

While diffusion models dominate the field of visual generation, they are computationally inefficient, applying a uniform computational effort regardless of different complexity. In contrast, autoregressive (AR) models are inherently complexity-aware, as evidenced by their variable likelihoods, but are often hindered by lossy discrete tokenization and error accumulation. In this work, we introduce Generative Refinement Networks (GRN), a next-generation visual synthesis paradigm that addresses these issues. At its core, GRN addresses the discrete tokenization bottleneck through a theoretically near-lossless Hierarchical Binary Quantization (HBQ), achieving a reconstruction quality comparable to continuous counterparts. Built upon HBQ's latent space, GRN fundamentally upgrades AR generation with a global refinement mechanism that progressively perfects and corrects artworks -- like a human artist painting. Besides, GRN integrates an entropy-guided sampling strategy, enabling complexity-aware, adaptive-step generation without compromising visual quality. On the ImageNet benchmark, GRN establishes new records in image reconstruction (0.56 rFID) and class-conditional image generation (1.81 gFID). We also scale GRN to more challenging text-to-image and text-to-video generation, delivering superior performance on an equivalent scale. We release all models and code to foster further research on GRN.

1 Introduction

GRN targets fixed-cost diffusion generation and error-prone autoregressive synthesis with adaptive computation, near-lossless discrete tokenization, and global refinement. It reports strong ImageNet results and extends to text-to-image and text-to-video generation.

  • Motivation: Diffusion models allocate identical computational resources to samples despite differing complexity, while standard autoregressive models suffer lossy discrete tokenization and accumulated multi-step errors.These limitations motivate a generation framework with adaptive computation and error correction.
  • GRN: GRN combines a global refinement mechanism with complexity-aware generation to overcome fixed diffusion costs and standard autoregressive shortcomings.The framework is designed to distribute computation according to visual difficulty while correcting earlier mistakes.
  • HBQ: HBQ provides near-lossless discrete image and video tokenization by ensuring exponentially decaying reconstruction error without increasing latent channels.It is reported to match continuous tokenizers at a higher compression rate.
  • Results: When scaled to text-to-image and text-to-video tasks, GRN demonstrates superior performance at equivalent scale and generates 1024×1024 images and 480p videos lasting 2 to 10 seconds.The text-to-video and text-to-image results demonstrate task generalization and scalability.

2 Related Work

Related work has improved visual tokenization, autoregressive generation, and diffusion efficiency, but each line retains important limitations. These limitations motivate more efficient quantization and adaptive visual generation.

  • Visual Tokenization: Vector-quantized visual tokenizers use discrete codebooks but retain a performance gap to continuous representations, while larger vocabularies increase convergence time and model size.These trade-offs motivate more efficient quantization schemes.
  • Visual Tokenization: Hierarchical Binary Quantization applies several binary quantization rounds to each VAE-encoded feature, with quantization error decaying exponentially across rounds.The figure text states that rapid lossless quantization is theoretically possible.
  • Autoregressive Generation: Autoregressive visual methods use next-token prediction, parallel decoding, or next-scale prediction, but remain limited by lossy tokenization and error accumulation.The cited passage also states that these models lag behind diffusion methods.
  • Adaptive Computation: Diffusion models typically require tens of inference steps, and distillation still relies on predefined fixed-step schedules that waste computation on simple prompts.AdaDiff addresses instance-specific steps using an external network and policy-gradient optimization.

3 Method

GRN combines near-lossless hierarchical binary quantization with autoregressive global refinement and entropy-guided sampling. Its tokenizer progressively represents visual features from coarse to fine, while generation fills, refines, and erases tokens using adaptive computation.

  • Visual Tokenizer: HBQ maps VAE features into binary tokens through repeated coarse-to-fine quantization rounds.Features are first bounded with tanh, then quantized using a binary tree of bucket centers.
  • Visual Tokenizer: Quantization error for round j is less than 1/2^j, so its upper bound decays exponentially with additional rounds.The resulting intermediate reconstructions exhibit a coarse-to-fine progression.
  • Visual Tokenizer: The quantized feature is reconstructed as a weighted sum of binary-round outputs and passed to the decoder during tokenizer training.Training uses a straight-through estimator so gradients reach the encoder, with reconstruction, perceptual, and GAN losses.
  • Generative Refinement Network: GRN begins from a random token map and repeatedly predicts drawing maps while refining all input tokens through a selection mask.The state combines current drawn tokens with random tokens; the mask proportion increases monotonically toward a complete token map.
  • Generative Refinement Network: The refinement loop can add confident tokens and erase errors as more context becomes available.The next-step drawing is predicted by a transformer conditioned on class embeddings or text, and the state is updated with a new selection map.
  • Training: GRN training constructs partially observed token maps by mixing ground-truth and random tokens, then predicts ground-truth tokens with cross-entropy.Sampling occurs across spatial, temporal, and channel dimensions without additional priors.
  • Complexity-Aware Sampling: Entropy-guided scheduling allocates fewer refinement steps to confident predictions and more steps to high-complexity generation.Entropy is averaged across token categories; GRNind uses K=2^M categories, while GRNbit uses K=2.

4 Experiments

Experiments evaluate GRN’s tokenizers and generation models across class-conditional image, text-to-image, and text-to-video synthesis, including refinement, sampling, and efficiency analyses.

  • 4.1 Visual Tokenizer: 0.56 rFID establishes state-of-the-art ImageNet 256×256 reconstruction with four HBQ rounds.This surpasses SD-VAE’s 0.87 rFID at a 4× higher compression rate and outperforms RAE, VAR, LlamaGen, and Open-MAGVIT-v2.
  • 4.1 Visual Tokenizer: Eight HBQ rounds achieve reconstruction quality nearly identical to the continuous baseline without increasing latent channels.Reconstruction metrics improve as HBQ rounds increase, while expanding channels from 16 to 64 raises PSNR from 30.40 to 33.97.
  • 4.2 Class-to-Image Results: 1.81 FID gives GRN-G state-of-the-art ImageNet class-conditional generation, while GRN-B reaches 3.56 FID with nearly half MaskGIT’s parameters.GRN-G also outperforms DiT, SiT, LlamaGen, and VAR on reported comparisons.
  • 4.3 Text-to-Image Results: 0.76 GenEval demonstrates that the 2B text-to-image model outperforms similarly sized SD3 Medium and Infinity models.Larger 6B–20B methods achieve higher scores, but GRN’s comparison is favorable at equivalent scale.
  • 4.4 Text-to-Video Results: GRN achieves higher VBench quality, semantic, and overall scores than several diffusion, flow-based, and autoregressive text-to-video models.The 2B model surpasses the larger CogVideoX-5B and comparable-size URSA, while InfinityStar-8B reports a higher overall score of 83.74.
  • 4.5.3 Complexity-Aware Sampling: Complexity-aware sampling reduces FID from 3.56 to 3.47 while providing up to 2.5× maximum and 1.25× average speedups.Samples receive 20–50 refinement steps, and over 97.9% require fewer than 50 steps.

5 Conclusion

GRN combines global refinement and complexity-aware generation with HBQ tokenization to address quantization loss and error accumulation. It reports state-of-the-art image results and superior equivalent-scale text-to-image and text-to-video performance.

  • GRN is a visual synthesis framework characterized by global refinement and complexity-aware generation.
  • HBQ develops discrete image and video tokenizers that are on par with continuous counterparts at the same latent dimensions and with higher compression.
  • GRN achieves state-of-the-art image reconstruction and class-conditional image generation, while surpassing equivalent-scale autoregressive and diffusion approaches in text-to-image and text-to-video generation.
  • GRN's global refinement mechanism addresses quantization loss and error accumulation that limited previous visual autoregressive models.

6 Limitations

The supplied passages primarily describe GRN's HBQ and iterative refinement procedures rather than author-stated limitations. They show coarse-to-fine binary quantization, mixed-token training, and sampling through filling, refining, and erasing.

  • HBQ procedure: HBQ orders binary tokens from coarse semantic concepts to fine high-frequency details across sequential quantization rounds.
  • HBQ procedure: The HBQ algorithm encodes VAE features, performs M binary quantization rounds, reconstructs a quantized feature with binary powers, and decodes it into an image or video.
  • Training: GRN training mixes ground-truth and random tokens, teaching the model to distinguish reliable from unreliable inputs while predicting the complete ground-truth token set.
  • Sampling: GRN sampling iteratively predicts complete token sets from partially drawn states, then updates them through filling, refining, and erasing.
  • Global refinement: Compared with fixed-order autoregressive models, GRN uses flexible global refinement so previously generated tokens can be revised rather than remaining immutable.

D Implementation Details

The implementation uses task-specific architectures and visual tokenizers across class-conditional image, text-to-image, and text-to-video generation. Sampling uses classifier-free guidance, while GRN's refinement process iteratively revises visual representations.

  • Model Architecture: The models cover C2I, T2I, and T2V tasks, including C2I variants from 130M to 2B parameters and a 2B-parameter architecture for T2I and T2V.
  • Visual Tokenizer: The C2I tokenizer uses 16 latent dimensions and four HBQ rounds to compress 256 × 256 images into binary tokens, achieving 0.56 rFID on ImageNet.
  • Training: C2I models train on 256×256 ImageNet images for 600 epochs, while T2I and T2V models use a coarse-to-fine training strategy across resolutions.
  • Sampling: GRN's global refinement mechanism iteratively revises and enhances the entire visual representation, mitigating error propagation in conventional autoregressive models.
  • Sampling: Classifier-free guidance is dynamically started for ImageNet models and applied throughout sampling for text-conditional models.

E.1 GAN Loss in Tokenizer

Increasing HBQ rounds improves reconstruction, while increasing GAN loss weight improves perceptual quality at a small reconstruction-fidelity cost. The tokenizer experiments therefore expose a quality trade-off controlled by these settings.

  • HBQ rounds: Reconstruction quality steadily improves as more HBQ rounds are introduced, with eight rounds nearly matching the unquantized baseline.
  • Experimental setup: The study compares tokenizer variants across different GAN loss weights and reports the associated perceptual-quality and reconstruction-fidelity changes.
  • GAN loss weight: Increasing λGAN from 0.001 to 0.02 reduces rFVD from 48.6 to 28.6 but lowers PSNR from 34.05 to 33.73.

E.2 Impacts of HBQ Rounds on Generation Performance

Generation performance depends on how many HBQ rounds distribute information across the latent representation and on the sampling and decoding choices used during refinement. Larger models benefit from more HBQ rounds, while random sampling and tuned decoding parameters affect generation behavior.

  • HBQ rounds: HBQ (M=2) performs best for GRN-B, but HBQ (M=4) surpasses it for GRN-L.The comparison uses GRN-B with 130M parameters and GRN-L with 458M parameters.
  • HBQ rounds: Larger models benefit from more HBQ rounds, although similar reconstruction metrics can still produce different generation metrics.The passage reports notable generation discrepancies between six and eight rounds despite nearly identical reconstruction metrics.
  • Bit prediction: MSB prediction is substantially more accurate than LSB prediction in the reported experiments.The passage identifies comparisons between HBQ configurations as a future research direction rather than reporting their results.
  • Sampling: Confidence-based sampling selects tokens deemed more correct for updating the next state, but the alternative produced inferior results.GRN constructs the binary selection map without prior constraints and randomly selects current predictions for refinement updates.
  • Decoding: For GRNbit-B, the reported optimal decoding settings are τ = 1.23, CFG = 2.4, and interval [0.44, 1].Increasing τ or decreasing CFG encourages diversity but can increase instability; the CFG interval helps restore diversity under higher CFG strength.

F.1 C2I Qualitative Results

GRN-G qualitative class-to-image samples are shown using the same decoding settings that achieved the reported FID of 1.81, while additional GRN-T2I results are presented at 1024×1024 resolution.

  • C2I qualitative results: GRN-G samples use CFG scale 1.7 and CFG interval [0.3, 1.0], matching the settings used to achieve FID 1.81.The visualization is intended to represent the model’s reported performance rather than use a higher CFG scale for appearance.
  • C2I qualitative results: GRN-T2I qualitative results are presented as 1024×1024 generated images.The passage identifies these as additional text-to-image results in Figure 14.

F.3 T2V Qualitative Results

The paper presents additional qualitative results for GRN-T2V, including videos depicting people performing actions in indoor environments. One example includes text in the video.

  • T2V qualitative results: Figures 15 and 16 present additional text-to-video generation results from GRN-T2V.The passages identify these figures as further qualitative results for the text-to-video task.
  • T2V qualitative results: One example shows a woman handling and hanging a striped skirt in a clothing-filled closet, with text appearing in the video.The described sequence includes movement, dialogue, object handling, and a change in the skirt’s location.
  • T2V qualitative results: Another example shows a man turning and walking forward through a kitchen with wooden cabinets and a tiled backsplash.The sequence follows the man as he turns right, moves forward, and passes the kitchen counter and cabinets.
Loading 2604.13030v2…