Generative Refinement Networks for Visual Synthesis

• Diffusion models lack adaptive-step capacity and allocate identical computational resources to samples of differing complexity.The paper attributes this fixed-step limitation to optimization by mean squared error without explicit likelihoods.Source: §1 · p. 2
• Standard AR models suffer from inferior reconstruction with discrete tokens and accumulated multi-step errors because causal or high-confidence predictions cannot be revised.Even masked AR approaches lack holistic refinement because high-confidence tokens become immutable.Source: §1 · p. 2
• GRN uses entropy-guided adaptive steps to allocate computation by visual difficulty and global refinement to retroactively correct accumulated generation errors.The framework is designed to overcome diffusion’s rigid computational cost and standard AR models’ shortcomings.Source: §1 · p. 2
• HBQ reduces reconstruction error exponentially without increasing latent channels, enabling discrete image and video tokenizers to match continuous tokenizers at higher compression.The paper introduces discrete tokenizers reported to be on par with continuous ones at the same latent dimensions.Source: §1 · p. 2
• GRN achieves state-of-the-art standard C2I results, with 0.56 rFID and 1.81 gFID, and outperforms equivalent-scale methods on T2I and T2V.The paper also reports scaling to photorealistic 1024×1024 images and high-fidelity 480p videos lasting 2–10 seconds.Source: §1 · p. 2

2.1 Visual Tokenizer

• Visual tokenizers progress from codebooks to lookup-free and large-vocabulary schemes, but scaling quality comes at costs in convergence speed and model size.Figure 2 depicts HBQ, whose quantization error decays exponentially over hierarchical rounds, theoretically enabling rapid lossless quantization.Source: §2.1 · p. 2, Figure 2 · p. 3

2.2 Autoregressive Models

• AR visual generation uses next-token, parallel-token, and next-scale strategies, yet remains constrained by lossy tokens and error accumulation.MaskGIT [7] generates high-confidence tokens before iteratively filling the remainder, while VAR [52] achieves over 10× faster inference; Infinity [26] assumes fewer than 30% diffuse errors, limiting covered patterns.Source: §2.2 · p. 3

2.3 Adaptive-step Generation

• Diffusion typically requires tens of inference steps, while distillation retains fixed schedules and AdaDiff adds a network and reward signals for instance-specific steps.Fixed-step schedules waste computation on simple prompts.Source: §2.3 · p. 3

3.1 Visual Tokenizer

• A Wan 2.1 3D causal VAE tokenizes images and videos into spatiotemporal features, which tanh bounds before hierarchical binary quantization splits values through a binary tree.The encoded dimensions are [1 + T, H/16, W/16] with C channels.Source: §3.1 · p. 3
• Each HBQ round adds a coarse-to-fine binary label, with round-j quantization error below 1/2^j and an exponentially decaying upper bound.Intermediate reconstructions illustrate the coarse-to-fine progression.Source: §3.1 · p. 4, Figure 3 · p. 4
• The decoder reconstructs images or videos from quantized features, while straight-through estimation trains the encoder with weighted reconstruction, LPIPS perceptual, and PatchGAN losses.For generation, GRNind encodes the M quantization rounds as integer tokens, whereas GRNbit concatenates the C and M dimensions into binary tokens; both predict channels in parallel.Source: §3.1 · p. 4

3.2 Generative Refinement Network

• GRN starts from a random token map and iteratively predicts drawing maps, selecting among current predictions and random blank tokens so the retained-token fraction increases toward a complete map.The transformer conditions its next-map prediction on the current state and generation condition; refinement can retain confident tokens and erase errors as more context accumulates.Source: §3.2 · pp. 4–5, Figure 4 · p. 5
• During training, GRN mixes uniformly sampled random tokens with ground-truth tokens under randomly sampled selection ratios, then predicts ground-truth tokens from partial inputs using cross-entropy.Sampling occurs across spatial, temporal, and channel dimensions without additional priors.Source: §3.2 · p. 5
• GRN schedules the retained-token fraction using average prediction entropy: low entropy yields fewer refinement steps and faster retention, while high entropy yields more steps and slower progression.A five-step warm-up precedes entropy-based scheduling; the resulting inference-step count is clipped to [Tmin, Tmax].Source: §3.2 · pp. 5–6

4.1 Visual Tokenizer

• 0.56 rFID: four-round HBQ outperforms the listed reconstruction baselines on 256×256 ImageNet, including continuous SD-VAE (0.87) at 4× higher compression.For joint image-video tokenization, eight-round HBQ nearly matches the continuous baseline without increasing latent channels; expanding channels from 16 to 64 raises PSNR from 30.40 to 33.97.Source: Figure 5 · p. 6, §4.1 · p. 7

4.2 Class-to-Image Results

• FID 1.81: GRN-G achieves state-of-the-art ImageNet 256×256 class-conditional generation, outperforming DiT, SiT, LlamaGen, and VAR.GRN-B has FID 3.56 versus MaskGIT’s 6.18 with nearly half the parameters; GRN-G also surpasses DiT and SiT in Inception Score.Source: §4.2 · p. 7, Table 3 · p. 8

4.3 Text-to-Image Results

• 0.76 overall on GenEval: GRN with a rewriter outperforms similarly sized SD3 Medium (0.62) and Infinity (0.71), despite larger methods scoring higher.The compared larger methods use 6B–20B parameters, while GRN uses 2B.Source: §4.3 · p. 8, Table 4 · p. 9

4.4 Text-to-Video Results

• GRN’s 2B text-to-video model scores higher than the listed diffusion, flow-based, and comparable-size URSA models, but trails 8B InfinityStar’s overall score of 83.74.GRN also outperforms the listed autoregressive counterparts; the paper reports that generated videos capture prompt semantics while maintaining aesthetic and visual quality.Source: §4.4 · pp. 8–9, Table 5 · p. 9

4.5.1 Predict Indices vs. Predict Bits

• FID 2.47 versus 2.64: GRN-L performs better when predicting bits rather than indices, while GRN-B slightly favors index prediction.The two prediction formats achieve comparable performance on class-to-image generation.Source: §4.5.1 · pp. 9–10, Table 6 · p. 9

4.5.2 Global Refinement Mechanism

• FID 3.63 versus 18.13: global refinement outperforms optimally tuned mask-based generation; with identical decoding settings, the mask-based approach collapses to FID 185.62.The comparison supports the paper’s claim that refinement mitigates error propagation by allowing previously generated tokens to be corrected.Source: §4.5.2 · p. 11, Table 7 · p. 11

4.5.3 Complexity-Aware Sampling

• FID 3.47 versus 3.56: complexity-aware sampling improves on fixed 50-step sampling while providing up to 2.5× and average 1.25× speedups.Across 63K images, steps vary from 20 to 50, over 97.9% use fewer than 50 steps, and 6.3% use the minimum; T2I sampling uses Tmin = 10.Source: §4.5.3 · pp. 11–12, Figure 9 · p. 11, Figure 8 · p. 11

4.5.4 Training and Inference Cost

• 4s average: GRN-2B generates a 1024×1024 T2I image in 2–8s, while GRN produces a 480p, 81-frame T2V video in 125s.Reported comparisons are Infinity 1s, FluxDev 12s, and Qwen-Image 33s for T2I; Wan2.1 480s and Wan2.2 437s for T2V. Tests used the same devices, but baselines used Diffusers optimizations while GRN ran in bare PyTorch eager mode.Source: §4.5.4 · p. 12

5 Conclusion

• At equivalent scales, GRN surpasses existing autoregressive and diffusion-based approaches in text-to-image and text-to-video generation, while setting state-of-the-art results in image reconstruction and class-conditional image generation.The conclusion provides no metric values for these comparisons.Source: §5 · p. 12
• HBQ produces discrete image and video tokenizers on par with continuous counterparts using the same latent channels and offering significantly higher compression.The paper presents HBQ as addressing quantization loss in visual autoregressive models.Source: §5 · p. 12
• GRN combines global refinement with complexity-aware generation and addresses quantization loss and error accumulation that limited prior visual autoregressive models.Source: §5 · p. 12
• The authors believe discrete-token GRN could integrate naturally with large language models, support unified text-and-visual learning, and become a strong competitor to Transfusion [70].They suggest unified learning could promote multimodal understanding and generation.Source: §5 · p. 12

6 Limitations

• GRN was not scaled to leading visual-generation models’ training compute or model size; its text-to-video outputs can lack visual detail and show distortions, especially outside human-related scenarios.The authors suggest balancing the data distribution and scaling model size could alleviate these limitations.Source: §6 · p. 12

B Algorithm for Generative Refinement Network

• GRN trains on hybrid feature maps containing ground-truth and random tokens, predicting all ground-truth tokens so it learns to preserve reliable tokens and refine unreliable ones.The input combines a subset of ground-truth tokens with a complementary subset of random tokens.Source: B Algorithm for Generative Refinement… · p. 16
• During sampling, GRN iteratively predicts a complete token set from the current state and randomly selects predictions to update it, unifying filling, refining, and erasing.The state combines already drawn content with random tokens in the remaining blank regions.Source: B Algorithm for Generative Refinement… · p. 16

C Difference with Other Autoregressive Models

• GRN revises the full visual representation through filling, refining, and erasing, unlike fixed-order GPT-style [50], VAR [52], and masked AR [7] models, whose generated tokens remain immutable.This flexible global refinement is described as mitigating error propagation in conventional autoregressive models.Source: C Difference with Other Autoregressive… · p. 17

Model Architecture

• C2I models scale from 130M to 2B parameters; T2I and T2V use a new 2B-parameter architecture meeting FlexAttention’s head-dimension requirement.Models support sequence packing and use NaViT [12] for arbitrary aspect ratios and resolutions; the C2I variants follow JiT [32].Source: D Implementation Details · p. 17

Visual Tokenizer

• The C2I tokenizer achieves reconstruction FID 0.56 on ImageNet, while the unified T2I/T2V tokenizer reaches FVD 26.3 and PSNR 34.73 on video reconstruction.The C2I tokenizer uses a 16-dimensional latent space and four HBQ rounds; the unified tokenizer uses 64 dimensions, spatial stride 16, temporal stride 4, and four HBQ rounds.Source: D Implementation Details · p. 17

Training

• C2I models train on 256×256 ImageNet images for 600 epochs, while T2I trains for 150K iterations at 256×256 and 60K at 1024×1024.C2I uses batch size 1024, learning rate 2e-4, and 10% condition dropping; T2I uses respective batch sizes 15400 and 2048, learning rates 2e-4 and 2e-5, zero weight decay, and 10% condition drop.Source: D Implementation Details · pp. 17–18

Sampling

• Benchmark sampling uses 50 fixed refinement steps, while complexity-aware C2I and T2I sampling tunes k over [300, 1200] and sets b to determine total steps.ImageNet CFG starts at an optimized threshold in [0, 0.5] with strength swept over [1.0, 3.0]; text-conditional sampling applies CFG throughout, searching strengths [1.0, 4.0] and temperature τ [0.5, 1.5].Source: D Implementation Details · p. 18

E.1 GAN Loss in Tokenizer

• Reconstruction quality steadily improves as more HBQ rounds are introduced, with four to six rounds sufficient for decent performance.The passage reports this trend in Fig. 5.Source: §E.1 · p. 18
• An eight-round HBQ model closes the reconstruction gap, achieving results nearly the same as the baseline without quantization.The passage reports this comparison in Fig. 5.Source: §E.1 · p. 18
• Increasing λGAN from 0.001 to 0.02 reduces rFVD from 48.6 to 28.6.This study fine-tuned an HBQ tokenizer with 64 latent channels and four quantization rounds.Source: §E.1 · p. 19, Table 9 · p. 19
• Increasing λGAN from 0.001 to 0.02 decreases PSNR from 34.05 to 33.73.This study fine-tuned an HBQ tokenizer with 64 latent channels and four quantization rounds.Source: §E.1 · p. 19, Table 9 · p. 19
• A λGAN weight of 0.005 strikes an effective balance between perceptual quality and reconstruction fidelity.This setting is applied to the GRN-T2I and GRN-T2V models.Source: §E.1 · p. 19
• The GAN-weight study fine-tunes a baseline HBQ tokenizer with 64 latent channels and four quantization rounds.It varies λGAN to examine its effect on reconstruction metrics.Source: §E.1 · p. 19

E.2 Impacts of HBQ Rounds on Generation Performance

• Larger GRN models benefit from more HBQ rounds: M=2 performs best for GRN-B (130M), while M=4 surpasses it for GRN-L (458M).From 6 to 8 rounds, reconstruction metrics remain nearly identical but generation metrics differ, indicating that distributing information across more bits requires greater model capacity.Source: §E.2 · p. 19, Table 10 · p. 19
• MSB accuracy is substantially higher than LSB accuracy; comparing HBQ (16dim, M=8) with HBQ (32dim, M=4) is proposed as future work.Source: §E.2 · p. 20

E.3 Binary Selection Map: Random vs. Confidence

• FID rises from 3.63 to 10.64 with confidence-based rather than random token selection, because confidence selection clusters updates and shifts inputs away from the training distribution.Training uses states with ground-truth and random tokens uniformly distributed; confidence-based selection violates that pattern.Source: §E.3 · p. 20, Table 11 · p. 20

E.4 Bit Prediction Target: Absolute vs. Relative

• Absolute bit prediction targets ground-truth bits, whereas relative prediction targets whether each input bit should flip, representing that decision as a residual.A relative target of 1 indicates a required flip; 0 indicates preservation.Source: §E.4 · p. 20

E.5 Decoding Hyper-Parameters

• Complexity-aware sampling outperforms linear and cosine schedules in fewer steps, with good results for k in [600, 800].The main tuning parameters are k for dynamic range and b for average steps; Tmin and Tmax clip step counts to a reasonable range.Source: §E.5 · p. 20, Table 12 · p. 20
• For GRNbit-B, optimal decoding parameters were τ = 1.23, CFG = 2.4, and CFG interval [0.44, 1]; higher τ or lower CFG encourages diversity but can increase instability.The interval disables CFG during initial decoding steps, which are crucial for overall semantics; many parameter combinations yield similar results.Source: §E.5 · p. 20, Figure 12 · p. 21

F.1 C2I Qualitative Results

• 256×256 uncurated GRN-G samples are shown in Fig. 13.The samples use CFG scale 1.7 and CFG interval = [0.3, 1.0], the settings used to achieve the best FID of 1.81.Source: §F.1 · p. 21
• The qualitative GRN-G examples use the same CFG settings as the best-FID evaluation rather than a higher CFG scale.The passage presents this choice as a way to make the visualization fair and representative of performance measured by FID.Source: §F.1 · p. 21

F.2 T2I Qualitative Results

• 1024×1024 images generated by GRN-T2I are shown in Fig. 14.Source: §F.2 · p. 21

• Figures 15 and 16 show additional text-to-video generation results from GRN-T2V.Source: §F.3 · p. 21, Figure 15 · p. 24, Figure 16 · p. 25