PaperlayerView exact revision on arXiv

Source-linked AI summary

Heptapod: Language Modeling on Visual Signals

Yongxin Zhu, Jiawei Chen, Yuanzhe Chen, Zhuo Chen, Dongya Jia, Jian Cong, Xiaobin Zhuang, Yuping Wang, Yuxuan Wang

arXiv:2510.06673v1cs.CVcs.AI

TL;DR

Directly transferring 1D next-token language modeling to visual signals remains challenging. Heptapod uses a causal Transformer with a reconstruction-focused tokenizer to predict a full 2D spatial distribution, achieving FID 2.70 versus 9.38 for LlamaGen-3B.

  • Problem

    Directly transferring 1D next-token language modeling to images remains challenging, motivating visual-generation approaches that incorporate external semantic guidance.

    Source: §1 · p. 1

  • Method

    Heptapod uses a causal Transformer and reconstruction-focused tokenizer to predict distributions over all subsequent positions in the 2D image grid.

    Source: §1 · pp. 2–3

  • Results

    FID 2.70 and IS 229.8 for Heptapod-H versus FID 9.38 and IS 112.9 for LlamaGen-3B, with less than one-third the parameters.

    Source: §5.4 · p. 10

  • Takeaways & Limitations

    Holistic 2D prediction bridges autoregressive generation and masked autoencoding while encouraging a holistic understanding of images.

    Source: §7 · p. 12

    Whether the visual-semantic principle extends to acoustic signals remains unclear; global prediction is computationally infeasible for extremely long or unbounded audio streams.

    Source: §6.2 · p. 12

Abstract

from arXiv · show

We introduce Heptapod, an image autoregressive model that adheres to the foundational principles of language modeling. Heptapod employs \textbf{causal attention}, \textbf{eliminates reliance on CFG}, and \textbf{eschews the trend of semantic tokenizers}. Our key innovation is \textit{next 2D distribution prediction}: a causal Transformer with reconstruction-focused visual tokenizer, learns to predict the distribution over the entire 2D spatial grid of images at each timestep. This learning objective unifies the sequential modeling of autoregressive framework with the holistic self-supervised learning of masked autoencoding, enabling the model to capture comprehensive image semantics via generative training. On the ImageNet generation benchmark, Heptapod achieves an FID of $2.70$, significantly outperforming previous causal autoregressive approaches. We hope our work inspires a principled rethinking of language modeling on visual signals and beyond.

1 Introduction

Directly transferring language-modeling principles from 1D text to 2D images has proven challenging, encouraging the use of external semantics. Heptapod instead uses a reconstruction-focused tokenizer and a causal Transformer trained to predict image-wide spatial distributions.

1 Introduction

  • Visual generation methods rely on external help, including inference-time CFG and SSL-derived semantic tokenizers, despite the language-modeling principle of learning semantics within the Transformer.The introduction describes performance degradation when CFG is disabled and when external DINO is removed.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • Heptapod asks whether faithful reconstruction can remain the tokenizer’s sole role while complex semantics emerge implicitly through next-token prediction.Its proposed separation assigns reconstruction to the tokenizer and semantic learning to the Transformer.Source: §1 · pp. 2–3
  • Heptapod predicts token distributions at every subsequent spatial position in parallel, extending next-token prediction to a holistic 2D target.The prefix supplies visible context, and the loss covers the remaining grid positions.Source: §1 · p. 2, Figure 2 · p. 3
  • Discrete token prediction uses a linear classification head with cross-entropy, while the diffusion-style head uses MSE for distribution modeling over visual representations.The two heads support the framework’s treatment of visual representations as distributions to predict.Source: §1 · pp. 2–3
  • The objective unifies causal autoregressive modeling with MAE-style learning: the prefix is unmasked context, while the full grid is predicted in parallel.The authors argue that this requires a holistic predictive representation of image structure and semantics.Source: §1 · p. 3

2 Related Work

Related work contrasts reconstruction-focused and semantic visual tokenizers, alongside autoregressive methods that often rely on CFG or depart from causal attention. Heptapod retains a reconstruction-based tokenizer and causal attention while changing the prediction objective.

2.1 Visual Tokenization

  • Reconstruction-focused tokenizers compress images faithfully, but autoregressive models trained on them often rely heavily on CFG for competitive generation.Semantic tokenizers distill external SSL representations, but can impair reconstruction and expose tension between reconstruction and semantic objectives.Source: §2.1 · p. 3
  • Heptapod keeps the tokenizer reconstruction-based and shifts visual-semantic learning to the Transformer through next-2D distribution prediction.This leaves the tokenizer focused on faithful compression and reconstruction.Source: §2.1 · pp. 3–4

2.2 Image Language Modeling

  • Earlier visual autoregressive models commonly need CFG, while MAR and VAR depart from causal attention; Heptapod retains causal attention and reformulates the prediction target.The cited discussion identifies MAR’s bidirectional attention and VAR’s non-causal attention within each scale.Source: §2.2 · p. 4

3 On the Nature of Visual Tokens and Their Predictive Modeling

Discrete and continuous visual tokens differ in how their distributions are parameterized, not in the underlying language-modeling objective. Their distinct loss parameterizations nevertheless affect training dynamics.

3 On the Nature of Visual Tokens and Their Predictive Modeling

  • For language modeling, discrete and continuous visual tokens are mathematically equivalent in objective: both model the next-token distribution conditioned on Transformer context.The token type determines how that conditional distribution is parameterized and optimized.Source: §3 · p. 4
  • Discrete tokens use a categorical softmax trained with cross-entropy, which directly maximizes sequence log-likelihood; continuous tokens use diffusion to model complex distributions.The diffusion objective typically applies MSE to noise or the original token and is described as equivalent to maximizing a log-likelihood lower bound.Source: §3 · p. 4
  • Different parameterizations affect training dynamics: cross-entropy gives sharper feedback, while diffusion averages its learning signal across noise levels and may slow convergence.The authors say experiments corroborate these training-dynamics differences.Source: §3 · p. 4

4 Next 2D Distribution Prediction

Visual next-token prediction is ambiguous because images lack a natural sequence, and local correlations can reward interpolation over global semantics. Heptapod keeps 1D causal attention but predicts distributions across the 2D image grid, using a prediction head to model spatial dependencies.

4.1 The Curse of Locality in Visual Autoregressive Models

  • Local token correlations can let visual autoregressive models favor interpolation over long-range dependencies, while semantic tokenizers encode distant semantic relationships in advance.The paper argues this workaround departs from learning semantics inside the Transformer.Source: §4.1 · p. 5, Figure 3 · p. 5

4.2 The 2D Prediction Objective

  • At each timestep, Heptapod predicts token distributions at all 2D grid positions from the causal prefix, requiring global structure rather than a designated next location.The objective expands the target from a 1D token vocabulary to a 2D position-and-token space while retaining 1D causal attention.Source: §4.1 · p. 5, §4.2 · p. 6

4.3 Architectures for 2D Prediction Head

  • A prediction head maps causal Transformer outputs to a full 2D spatial distribution, with global and local variants trading modeling scope against computational cost.The global head uses bidirectional attention over the full grid; the local head uses cross-attention and bidirectional attention within a chunk.Source: §4.3 · pp. 6–7, Figure 5 · p. 7
  • Both head variants let one 1D causal step trigger a comprehensive 2D spatial prediction within standard autoregressive training.The paper presents this mechanism as prompting holistic understanding of visual data.Source: §4.3 · p. 7

5 Experiments

On ImageNet 256 × 256, Heptapod-H outperforms LlamaGen-3B while using fewer than one-third as many parameters. Experiments compare reconstruction-focused tokenizers, prediction-head designs, and supervision density, with all reported results obtained without CFG.

5.1 Experimental Setup

  • Experiments use ImageNet-1K at 256 × 256 with CFG disabled and evaluate FID and IS; both tokenizers encode images as 256-token latent grids.The VQGAN tokenizer has rFID 2.19 and the VAE tokenizer rFID 1.43.Source: §5.1 · pp. 7–8

5.2 Convergence Efficiency with Different Tokenizers

  • The VQ-tokenizer model converges faster initially, but the VAE-tokenizer model ultimately achieves superior generative performance, and tokenizer reconstruction quality sets a practical upper bound.The authors adopt the VAE tokenizer for later experiments; tokenizer and loss choice also affect training efficiency.Source: §5.2 · p. 8, Figure 6 · p. 8

5.3 Properties of Next 2D Distribution Prediction

  • Heptapod’s 2D-random objective outperforms 1D-random and 1D-raster, while full-window w256n1 outperforms smaller windows even with denser supervision.The table compares distribution samplers and generation orders; the window ablation reports FID and IS trends.Source: §5.3 · p. 9, Table 1 · p. 9, Figure 7 · p. 9

5.4 Benchmarking with Previous Generative Models

  • FID 2.70 and IS 229.8 for Heptapod-H outperform LlamaGen-3B (FID 9.38, IS 112.9) with less than one-third of its parameters.Reported results use a reconstruction-focused tokenizer and no CFG; Table 2 covers ImageNet 256 × 256 class-conditional generation.Source: §5.4 · p. 10, Table 2 · p. 10

5.5 Scale Supervision Density

  • With a global prediction window, increasing supervision density consistently improves FID and IS, but raises computational cost.The paper contrasts this with small windows, where denser supervision alone provides negligible gains.Source: §5.5 · p. 11, Figure 8 · p. 11

6 Discussion and Future Work

The discussion relates Heptapod’s next 2D distribution prediction to multi-token prediction, describes an open question about applying the approach to audio, and notes that its prediction principle is not tied to a particular visual tokenizer.

6.1 Connection with Multi-token Prediction

  • Heptapod extends multi-token prediction to images by predicting the entire remainder of the token grid, addressing the ambiguity of choosing a “next” position in 2D.The discussion argues that predicting all future locations requires global structure and semantics rather than relying on local interpolation.Source: §6.1 · p. 12

6.2 Language Modeling on Acoustic Signals

  • For audio, global prediction may be computationally infeasible in long or unbounded streaming settings, while a chunk-based head limits prediction to a finite local window.Whether the visual principle of holistic prediction would also hold for audio remains unclear.Source: §6.2 · p. 12

6.3 Tokenizer in Heptapod

  • Heptapod’s prediction head models distributions over 2D positions whether they represent discrete tokens or continuous patches, making the framework agnostic to token form.Fractal Generative Models [26] are cited as an example of high-fidelity pixel-space generation that bypasses tokenization.Source: §6.3 · p. 12

7 Conclusion

The conclusion presents next 2D distribution prediction as a way to extend autoregressive modeling beyond sequential data and combine aspects of autoregressive generation and masked autoencoding.

7 Conclusion

  • Heptapod decouples reconstruction from semantic learning and extends autoregressive modeling beyond sequential data through next 2D distribution prediction.The conclusion says this encourages holistic image understanding by predicting the 2D distribution.Source: §7 · p. 12
  • The approach bridges autoregressive generation and masked autoencoding, combining strengths the conclusion associates with both paradigms.Source: §7 · p. 12
  • The conclusion says the framework resolves ambiguity in visual next-token prediction and lays a foundation for unified generative and understanding capabilities across modalities.Source: §7 · p. 12

A Model Configuration

The appendix lists continuous-tokenizer configurations for three causal Transformer scales and reports that a balanced division of depth between the Transformer and prediction head performs best among the tested allocations.

A Model Configuration

  • For continuous tokenizers, the Base, Large, and Huge models use denoising MLP heads with 6, 8, and 12 blocks, respectively, and widths 1024, 1280, and 1536.The configuration follows MAR [25].Source: A Model Configuration · p. 19
  • A 16/16 split between the causal Transformer and 2D prediction head achieves the best FID and IS among the tested Heptapod-L allocations.The 24/8 split degrades performance, while 31/1 causes a sharp drop in both metrics.Source: A Model Configuration · p. 19
  • The reported allocation results indicate that the prediction head needs sufficient depth to project 1D causal context into 2D distributions and model position-wise interactions at scale.Source: A Model Configuration · p. 19

B Visualization of Tokenizer and Attention

The appendix figures show cosine-similarity visualizations of semantic and VQ-VAE tokens, along with attention-map visualizations for each tokenizer.

B Visualization of Tokenizer and Attention

Related papers

Loading 2510.06673v1…