PaperlayerView exact revision on arXiv

Source-linked AI summary

HART: Efficient Visual Generation with Hybrid Autoregressive Transformer

Haotian Tang, Yecheng Wu, Shang Yang, Enze Xie, Junsong Chen, Junyu Chen, Zhuoyang Zhang, Han Cai, Yao Lu, Song Han

arXiv:2410.10812v1cs.CVcs.AIcs.LG

TL;DR

Discrete tokenizers in AR models reconstruct significantly worse than continuous tokenizers in diffusion models, leaving AR models with a lower generation upper bound and difficulty modeling fine image details. HART decomposes continuous autoencoder latents into discrete structure tokens modeled by a scalable-resolution AR transformer and continuous residual tokens modeled by lightweight residual diffusion. HART closely matches state-of-the-art diffusion models on multiple text-to-image metrics, with 4.5–7.7× higher throughput and 6.9–13.4× less computation.

  • Problem

    Discrete tokenizers in AR models reconstruct significantly worse than continuous tokenizers in diffusion models, leaving AR models with a lower generation upper bound and difficulty modeling fine image details.

    Source: §1 · p. 3

  • Method

    HART decomposes continuous autoencoder latents into discrete structure tokens modeled by a scalable-resolution AR transformer and continuous residual tokens modeled by lightweight residual diffusion.

    Source: §1 · p. 3

  • Results

    HART closely matches state-of-the-art diffusion models on multiple text-to-image metrics, with 4.5–7.7× higher throughput and 6.9–13.4× less computation.

    Source: §1 · p. 3

  • Takeaways & Limitations

    HART directly generates 1024×1024 images from text prompts without super-resolution, with quality comparable to diffusion models.

    Source: §5 · p. 10

Abstract

from arXiv · show

We introduce Hybrid Autoregressive Transformer (HART), an autoregressive (AR) visual generation model capable of directly generating 1024x1024 images, rivaling diffusion models in image generation quality. Existing AR models face limitations due to the poor image reconstruction quality of their discrete tokenizers and the prohibitive training costs associated with generating 1024px images. To address these challenges, we present the hybrid tokenizer, which decomposes the continuous latents from the autoencoder into two components: discrete tokens representing the big picture and continuous tokens representing the residual components that cannot be represented by the discrete tokens. The discrete component is modeled by a scalable-resolution discrete AR model, while the continuous component is learned with a lightweight residual diffusion module with only 37M parameters. Compared with the discrete-only VAR tokenizer, our hybrid approach improves reconstruction FID from 2.11 to 0.30 on MJHQ-30K, leading to a 31% generation FID improvement from 7.85 to 5.38. HART also outperforms state-of-the-art diffusion models in both FID and CLIP score, with 4.5-7.7x higher throughput and 6.9-13.4x lower MACs. Our code is open sourced at https://github.com/mit-han-lab/hart.

1 INTRODUCTION

The introduction frames a gap between diffusion models’ image quality and efficiency and AR models’ reconstruction quality and high-resolution capability. HART combines discrete structure tokens with continuous residuals to target efficient, detailed 1024px generation.

1 INTRODUCTION

  • Discrete AR tokenizers struggle with fine details, while the introduction says no existing AR model directly and efficiently generates 1024×1024 images.Diffusion models offer better generation quality but can be computationally expensive; generating at 1024×1024 with DiT-XL/2 requires 86.2T MACs.Source: §1 · p. 3
  • HART decomposes autoencoder latents into discrete tokens for overall structure and continuous residual tokens for fine details, modeling them with a scalable VAR transformer and lightweight residual diffusion.The residual diffusion module adds 5% parameter and 10% runtime overhead.Source: §1 · p. 3, Figure 3 · p. 4
  • On MJHQ-30K at 1024×1024, HART reduces reconstruction FID from 2.11 to 0.30 and generation FID from 7.85 to 5.38, a 31% relative improvement over VAR.The introduction also reports up to 7.8% FID improvement over VAR for class-conditioned ImageNet generation and 13× higher throughput than MAR on that task.Source: §1 · p. 3
  • HART closely matches state-of-the-art diffusion models on multiple text-to-image metrics, with 4.5–7.7× higher throughput and 6.9–13.4× less computation.It also achieves 3.1–5.9× faster inference latency than diffusion models.Source: §1 · p. 3

2 RELATED WORK

Related work traces diffusion and autoregressive image generation as major approaches, alongside hybrid methods that combine discrete and continuous modeling. It also situates HART among work on tokenization, sampling speed, resolution, and multimodal generation.

2 RELATED WORK

  • Earlier image-generation work includes VAEs and GANs, with GAN improvements by Brock et al. (2018), Karras et al. (2019), and Kang et al. (2023).The passage presents these as predecessors to later diffusion and autoregressive approaches.Source: §2 · p. 4
  • Diffusion work progressed from latent U-Nets to transformer-based DiT and U-ViT, then scaled to text-to-image generation and large models.Recent work also explored rectified flow for fast sampling.Source: §2 · p. 4
  • Autoregressive image generation spans pixel sequences, VQ tokenizers, faster sampling, and VAR’s next-scale prediction; related work also includes text-conditioned and controllable generation.Concurrent work Liu et al. (2024a) achieves high-quality 1024px generation through AR modeling, while the paper says HART is more efficient.Source: §2 · p. 4
  • Hybrid approaches include autoregressive prediction of continuous tokens, diffusion over discrete latents, and diffusion conditioned on an autoregressive prior (Tschannen et al., 2023).Transfusion and DART are discussed as concurrent approaches to multimodal generation (Zhou et al., 2024; Gu et al., 2024).Source: §2 · p. 4

3 HART: HYBRID AUTOREGRESSIVE TRANSFORMER

HART combines a hybrid tokenizer with an autoregressive transformer and residual diffusion to model both discrete and continuous image tokens. The section describes how the tokenizer is trained, how the two token components are modeled, and measures to reduce training and inference overhead.

3 HART: HYBRID AUTOREGRESSIVE TRANSFORMER

  • HART models both discrete and continuous tokens with a unified transformer, using a hybrid tokenizer and residual diffusion as its key components.The framework is presented as combining discrete and continuous image modeling rather than operating exclusively in one latent space.Source: §3 · p. 5

3.1 HYBRID VISUAL TOKENIZATION

  • Hybrid tokenization quantizes encoder-produced continuous visual features across multiple scales and treats the difference between those features and accumulated discrete features as residual tokens.The residual tokens are modeled by residual diffusion; the tokenizer is intended to decode continuous features during generation.Source: §3.1 · p. 5
  • The hybrid tokenizer initializes from a pretrained discrete VAR tokenizer, freezes its encoder and quantizer, and trains the decoder with discrete or continuous inputs chosen equally often.The continuous path bypasses the VQ quantizer; the discrete path trains the standard VQ tokenizer.Source: §3.1 · pp. 5–6
  • Alternating training yields continuous rFID comparable to the SDXL tokenizer and discrete rFID matching the original VQ tokenizer, while keeping the continuous and discrete latents similar from the decoder’s perspective.The authors caution that low rFID alone does not necessarily indicate better generation FID; separate decoders can compromise generation FID despite similar continuous reconstruction FID.Source: §3.1 · p. 6

3.2 HYBRID AUTOREGRESSIVE MODELING WITH RESIDUAL DIFFUSION

  • HART represents continuous image tokens as discrete tokens modeled by a scalable-resolution AR transformer plus residual tokens modeled by residual diffusion.The final representation is their sum; the transformer supports text conditioning and generation up to 1024px.Source: §3.2 · p. 6, Figure 6 · p. 6
  • HART directly generates 1024px images with one model and fine-tunes from low-resolution checkpoints to mitigate the O(n^4) training cost of high-resolution AR transformers.Its relative position embeddings use sinusoidal step embeddings and a hybrid of 1D rotary embeddings for text and 2D rotary embeddings for visual tokens.Source: §3.2 · p. 6
  • HART’s 37M-parameter residual diffusion MLP models residuals in 8 inference steps, compared with MAR’s 30–50 steps, reducing diffusion-module overhead by 4–6×.The MLP is conditioned on the AR transformer’s last-layer hidden states and discrete tokens; HART models residuals rather than MAR’s full continuous tokens.Source: §3.2 · pp. 6–7

3.3 EFFICIENCY ENHANCEMENTS

  • Training on subsampled final-step tokens, discarding 80% on average, accelerates training by 1.4× at 512px and 1.9× at 1024px; kernel fusion improves end-to-end execution time by 7% and total runtime by 10%.The subsampling approach also reduces training memory usage by 1.1× without reported performance degradation.Source: §3.3 · p. 7

4 EXPERIMENTS

HART is evaluated on class- and text-conditioned generation, with results covering tokenization quality, benchmark performance, efficiency, and design ablations. Its hybrid tokenizer and training choices improve generation while retaining strong quality at higher resolutions.

4.1 SETUP

  • Experiments use 600M, 1B, and 2B class-conditioned transformers; text-conditioned models start from the 1B model, and evaluation spans ImageNet, MJHQ-30K, GenEval, and DPG-Bench.The tokenizer is trained on OpenImages; transformer training uses ImageNet, JourneyDB, and internal synthetic data, with latency and throughput measured on A100 GPUs.Source: §4.1 · pp. 7–8

4.2 MAIN RESULTS

  • At 1024×1024, HART achieves 4.5–7.7× higher throughput and 3.1–5.9× lower latency than state-of-the-art diffusion models.At 512×512, the corresponding gains are 5.0–9.6× throughput and 4.0–4.7× lower latency.Source: Table 3 · p. 8, §4.2 · p. 9
  • At 1024px, hybrid tokenization reduces rFID from 2.11 to 0.30 on MJHQ-30K, matching SDXL tokenizer performance while retaining discrete tokens that capture most image structure.The evaluations use ImageNet and MJHQ-30K, which were not seen during tokenizer training.Source: §4.2 · p. 8, Table 1 · p. 8
  • On MJHQ-30K, HART achieves better FID than diffusion models and better image-text alignment than 3.6× larger SD-XL; on GenEval and DPG-Bench, it is comparable to diffusion models under 2B parameters.Compared with SDXL, HART also has superior quality across all Table 2 benchmarks.Source: §4.2 · pp. 8–9, Table 2 · p. 8
  • HART outperforms MAR-L in class-conditioned FID and inception score with 10.7× fewer MACs and 12.9× lower latency; across model sizes, FID improves 4.3–7.8%.For larger models (d ≥24), residual diffusion accounts for 4–11% of runtime; HART’s largest model is 3.3× faster than DiT-XL/2 with a 20-step sampler.Source: §4.2 · p. 9, Table 4 · p. 9

4.3 ABLATION STUDIES AND ANALYSIS

  • Residual diffusion improves ImageNet 256×256 FID by 10–14% versus HART-VAR; on MJHQ-30K, its FID gains are 11.1% at 256px, 17.0% at 512px, and 31% at 1024px.It also provides a 6.1% FID improvement over the stronger discrete-only HART tokenizer.Source: §4.3 · pp. 9–10, Table 5 · p. 9
  • Learning residual tokens requires 3 diffusion sampling steps versus MAR’s 60 denoising steps, yielding a 20× reduction in runtime for continuous-token learning while achieving higher inception score.The figure comparison reports similar reconstruction FID for the single-decoder alternating-training approach and separate decoders, with faster and better generation convergence for the former.Source: §4.3 · p. 10, Figure 7 · p. 10
  • Replacing VAR’s absolute positional embeddings with relative embeddings improves convergence when fine-tuning HART at higher resolutions from low-resolution checkpoints.Doubling output resolution increases token count 4× and attention computation 16×.Source: §4.3 · p. 10, Figure 8 · p. 10

5 CONCLUSION

HART directly generates 1024×1024 images from text without super-resolution, with quality comparable to diffusion models and higher throughput. It combines discrete tokens for overall structure with residual tokens for image details.

5 CONCLUSION

  • HART directly generates 1024×1024 images from text prompts without super-resolution, with quality comparable to diffusion models and 3.1–5.9× higher speed.It offers 4.5–7.7× higher throughput.Source: §5 · pp. 10–11
  • Hybrid tokenization decomposes continuous image latents into discrete tokens for overall structure and residual tokens that refine details.A 37M-parameter residual diffusion module learns the residual tokens using 8 sampling steps.Source: §5 · pp. 10–11

A APPENDIX

The appendix figures contrast direct high-resolution generation with low-resolution generation and visualize attention patterns in later stages. The attention display includes a sigmoid transformation for clarity in its rightmost panels.

A APPENDIX

  • Figure 9 visualizes attention at stages 8–10, showing concentration on recent and initial stages and predominantly local attention within the final stage.The rightmost three subfigures apply a sigmoid function to attention scores for clearer visualization.Source: Figure 9 · p. 16
  • Figure 10 reports that direct 1024×1024 generation produces more detailed results than 512×512 generation.Source: Figure 10 · p. 16

A.1 ATTENTION PATTERN ANALYSIS

The attention analysis finds that VAR attends mainly to the current and preceding stages and the first three stages, with local attention within the current stage. This pattern helps explain why subsampling final-stage tokens during training does not degrade generation quality.

A.1 ATTENTION PATTERN ANALYSIS

  • Attention scores concentrate on the current stage, preceding stage, and initial three stages; within the current stage, each token primarily attends to immediate neighbors.This local pattern resembles convolution operations.Source: §A.1 · p. 16
  • Reducing final-stage tokens by up to 80% during training preserves the fundamental attention pattern, explaining why partial supervision does not compromise generation quality.Source: §A.1 · p. 16
  • Restricting training attention to the first three stages plus two local stages does not affect final results, making a sparse attention kernel a feasible future acceleration.Source: §A.1 · p. 16

Related papers

Loading 2410.10812v1…