PaperlayerView exact revision on arXiv

Source-linked AI summary

FeaTok: Feature-Loss Tokenization for Image Generation

Zhang Xiaofeng, Samuel Lavoie, Nicolas Beltran-Velez, Jianhao Yuan, Felix Friedrich, Jakob Verbeek, Aaron Courville, Michal Drozdzal, Adriana Romero-Soriano

arXiv:2610.22060v1cs.CV

TL;DR

Discrete image tokenizers are commonly trained for pixel reconstruction, although image generation also depends on tokens that generators can model easily. FeaTok trains discrete tokens by reconstructing pretrained visual features rather than pixels. It improves generation across reported settings, including a 29% gFDr6 reduction over the strongest discrete baselines without guidance at 256×256.

  • Problem

    Pixel-focused tokenizers prioritize reconstruction, while image generation also requires tokens that are easy for a generator to model.

    Source: §1 · p. 2

  • Method

    FeaTok trains discrete tokens by reconstructing pretrained visual features, with pixel losses applied only to a separate decoder trained on detached tokens.

    Source: §1 · p. 2

  • Results

    29% without guidance: FeaTok improves on the strongest discrete baselines in gFDr6 at 256×256 and advances the quality–compute Pareto front.

    Source: §1 · p. 2

  • Takeaways & Limitations

    FeaTok’s feature-shaped discrete tokens are competitive with continuous representations and can serve as a practical substrate for unified multimodal models.

    Source: §1 · p. 2

Abstract

from arXiv · show

Discrete image generators learn the distribution of tokens produced by an image tokenizer. These token representations should be easy for generators to model, not merely enable faithful image reconstruction. Yet these tokenizers are commonly trained to reconstruct pixels, which prioritizes low-level fidelity and may disrupt representation structure useful for generation. Motivated by feature-predictive self-supervised learning, we introduce feature-loss tokenization (FeaTok), which trains discrete tokens by reconstructing pretrained features rather than pixels. Ablations across SigLIP, SigLIP2, DINOv2, and DINOv3 show better generative performance with FeaTok. In class-conditional generation, FeaTok advances the quality-compute Pareto frontier, reducing gFDr6 at 256x256 by up to 29 percent over the strongest discrete baselines and outperforming the strongest unguided continuous baseline (RAEv2) with 38 times less sampling compute. FeaTok also achieves higher human-preference reward-model scores than prior art without reward-based training and surpasses continuous and discrete text-conditional baselines on three benchmarks with only 500 fine-tuning updates after pre-training.

1 Introduction

Discrete image generation needs tokens that are both expressive and easy for generators to model, but pixel-focused training prioritizes reconstruction. FeaTok instead shapes tokens with pretrained-feature reconstruction and achieves strong generation results.

1 Introduction

  • Image-generation tokens need both expressivity and generative learnability, meaning generators must be able to model their distribution.These properties do not always align.Source: §1 · p. 2
  • Representation structure is related to generative learnability, while pixel reconstruction prioritizes high-variance details over perceptually useful features.Self-supervised feature prediction yields more semantically meaningful representations than pixel prediction.Source: §1 · p. 2
  • Although prior work used feature reconstruction for discrete tokens, subsequent tokenizers retained pixel reconstruction as the central loss, prompting the question of whether it is essential.Discrete tokens also share the interface of language models and sample images with an order of magnitude less compute than continuous diffusion models.Source: §1 · pp. 1–2
  • FeaTok uses a feature decoder to reconstruct pretrained visual features, while a separate pixel decoder renders images from detached tokens.Its global representation is discretized and generated before patch tokens, providing them image-level context.Source: §1 · p. 2
  • 29% without guidance and 25% with guidance: FeaTok improves on the strongest discrete baselines in gFDr6 at 256×256.It also advances the quality–compute Pareto front and outperforms recent continuous generators at matched sampling cost in both settings.Source: §1 · p. 2

2 Feature-loss tokenization

FeaTok trains discrete tokens to reconstruct pretrained visual features rather than pixels, while retaining pixel decoding without allowing pixel losses to shape the tokens. Across four pretrained vision encoders, feature targets, global-token conditioning, and four-layer targets improve generation in the reported ablations.

2.1 Learning discrete tokens with feature loss only

  • FeaTok trains the encoder, quantizer, and feature decoder to reconstruct pretrained visual features using a target averaged from four normalized layers of a frozen vision encoder.The target is intended to combine earlier-layer visual detail with later-layer semantic information.Source: §2.1 · p. 3
  • Pixel reconstruction remains available for image decoding, but stopped gradients ensure its loss does not shape the discrete token distribution.The pixel loss updates the pixel decoder using detached patch tokens.Source: §2.1 · p. 3, Figure 2 · p. 3
  • FeaTok discretizes the encoder’s global representation and models it before patch tokens, providing image-level context for patch generation.The generator models the global token and patch tokens conditioned on it and the external condition.Source: §2.1 · pp. 3–4, Figure 2 · p. 3

2.2 Controlled ablations across pretrained vision encoders

  • Feature targets lower gFID for SigLIP, DINOv2, and DINOv3; global-augmented generation improves gFID for all four encoders, and four-layer-average targets perform best for each encoder.The feature target raises adjacent-patch dependence for SigLIP, DINOv2, and DINOv3; SigLIP2 is the exception to the patch-only feature-target improvement.Source: Figure 3 · p. 4, §2.2 · p. 5, Figure 4 · p. 5

3 State-of-the-art image generation with discrete tokens

FeaTok is evaluated for class-conditional ImageNet generation at 256×256 and 512×512, and for text-to-image generation. It outperforms discrete baselines, is competitive with continuous models, and surpasses text-to-image baselines after fine-tuning.

3 State-of-the-art image generation with discrete tokens

  • FeaTok achieves state-of-the-art performance among discrete methods at both ImageNet resolutions, outperforms continuous generators at matched inference cost at 256×256, and beats continuous and discrete text-to-image baselines after fine-tuning.The text-to-image evaluation covers GenEval, DPG-Bench, and GenAI-Bench.Source: §3 · p. 5

3.1 ImageNet-1k generation at 256×256

  • 3.13 gFDr6 for unguided FeaTok-DINOv3, versus 4.38 for VFMTok2-XXL; guided FeaTok scores 3.08 versus BAR-L’s 4.11.All four FeaTok-L variants outperform the strongest prior discrete baseline in gFDr6.Source: §3.1 · p. 6, Table 1 · p. 6
  • 3.13 gFDr6 at 1.27 TFLOPs per image for unguided FeaTok-L-DINOv3, versus RAEv2’s 3.95 at 47.72 TFLOPs.With guidance, FeaTok scores 3.08 and is competitive with RAE-XL at 3.26 and REPA-E at 3.04.Source: §3.1 · p. 6, Table 2 · p. 7
  • FeaTok achieves higher human-preference rewards than baselines and scores closer to training-data rewards across the reported preference measures.On HPSv3, FeaTok-DINOv3 and FeaTok-SigLIP are closest to the training-set scores, followed by RAEv2.Source: §3.1 · p. 7, Figure 5 · p. 8

3.2 FeaTok outperforms BAR-L and RAE at 512×512

  • 2.17 unguided gFDr6 and 1.80 guided gFDr6 for FeaTok-L-DINOv3 at 512×512, versus BAR-L’s 3.82 and 2.94.FeaTok also beats RAE in gFDr6, though RAE is better in the Inception and SigLIP2 spaces in both settings; RAEv2 has no 512-resolution model.Source: §3.2 · p. 7, Table 4 · p. 8

3.3 Extension to text-to-image generation

  • After 500 fine-tuning updates, FeaTok exceeds RAEv2 on GenEval (0.837 versus 0.827), GenAI-Bench (0.698 versus 0.680), and DPG-Bench (83.4 versus 82.3).The evaluation reuses the ImageNet-trained FeaTok-SigLIP2 tokenizer and compares against models pre-trained on the same total number of examples.Source: §3.3 · p. 8

4 Related work

Related work studies pretrained visual features in continuous and discrete image generation. FeaTok differs from prior discrete tokenizers by using only pretrained visual features to shape tokens, including a multi-layer target and a quantized global token.

Pretrained representations for continuous generation

  • Continuous-generation approaches use pretrained feature spaces, feature alignment, or joint denoising and latent alignment, with reported differences across feature targets.MAE features reconstruct best but generate worse than DINOv2 features.Source: §4 · pp. 8–9

Text-to-image results

  • The text-to-image comparison reports that FeaTok is broadly comparable to RAEv2 after pre-training and surpasses all baselines after 500 fine-tuning updates.Continuous-latent baseline results are taken from Singh et al. (2026).Source: Table 5 · p. 9

Discrete tokenization with pretrained representations

  • Discrete tokenizers have used pixel reconstruction, feature reconstruction, and added feature-based objectives; FeaTok uses only pretrained visual features to shape tokens.FeaTok also reconstructs a multi-layer target and includes a quantized global token.Source: §4 · p. 9

5 Conclusion

FeaTok separates feature-shaped token learning from pixel-based image rendering and also models a discretized global representation. The paper reports generation gains across four pretrained encoders, strong ImageNet results, higher human-preference rewards, and text-to-image transfer after 500 fine-tuning updates.

5 Conclusion

  • FeaTok tokens are shaped only by feature reconstruction, while a pixel decoder trained on detached tokens renders them into images.The method also discretizes the encoder’s global representation and models it together with patch tokens.Source: §5 · p. 9
  • Across four pretrained encoders, FeaTok generates better than tokenizers trained with pixel or combined losses and achieves the best evaluated discrete-generator gFDr6 on ImageNet at both resolutions.It also outperforms continuous generators at matched sampling cost and achieves higher human-preference reward-model scores.Source: §5 · p. 9
  • FeaTok transfers to text-to-image generation after pre-training and only 500 fine-tuning updates.The paper presents its low sampling cost as attractive for scaling multimodal training.Source: §5 · p. 9

A Tokenizer implementation details

FeaTok discretizes pretrained encoder features into patch and global tokens, trains their representation with feature losses, and restricts pixel-branch gradients to a separate rendering decoder.

A Tokenizer implementation details

  • For 256 × 256 tokenizers, a trainable projector maps patch features to 16 scalar latents that are quantized coordinatewise into 16-bit tokens.The bit vector indexes one of 2^16 patch tokens.Source: A Tokenizer implementation details · p. 14
  • The global quantizer prepends a 16-bit code for the encoder’s global feature, which the generator samples first to provide image-level context to patch tokens.The feature decoder reconstructs both global and patch targets; the pixel decoder consumes only sampled patch tokens.Source: A Tokenizer implementation details · p. 14
  • The feature decoder is matched to the encoder, while the pixel decoder follows BAR’s ViT-L design and is shared across encoders.At 512 × 512, a separate DINOv3-L tokenizer processes a 32 × 32 grid of patch tokens, each carrying 10 bits.Source: A Tokenizer implementation details · pp. 14–15
  • Feature and regularization losses train the encoder, input projector, and feature decoder, while stop-gradient confines pixel-branch updates to the pixel decoder and output projector.FSQ uses fixed coordinate levels and therefore needs no commitment or codebook loss.Source: A Tokenizer implementation details · p. 14

B Generator implementation details

FeaTok retains BAR’s masked-bit generator and adds a sampled global token for image context; ablations use fixed unguided settings, while MAE-based tokenizers perform poorly under every tested supervision.

B Generator implementation details

  • BAR’s masked-bit objective progressively predicts masked token bits, conditioning on visible bits and autoregressive context; the global token uses a separate masking-ratio schedule.The combined objective weights global and patch losses, and its normalization makes patch loss a bit-weighted rather than uniform average over positions.Source: B Generator implementation details · pp. 15–16
  • FeaTok retains BAR’s generator and masked-bit prediction, prepending a sampled image-dependent global token before patch tokens; 256 × 256 patch tokens contain 16 bits.At 512 × 512, the generator models one global token followed by 1,024 patch tokens of 10 bits each.Source: B Generator implementation details · pp. 16–17

C.1 Ablation protocol and sampling without guidance

  • Ablations compare tokenizer choices on 50K unguided samples with temperature 2.0 and allocation [4, 4, 4, 4], avoiding guidance as an extra sampling variable.Global-augmented variants sample the global token first, then condition patch generation on it.Source: §C.1 · pp. 17–18

C.2 Tokenizers from an MAE encoder

  • MAE-based tokenizers generate poorly under every supervision: patch-only gFID ranges from 5.74 to 6.40, and global-augmented gFID is 5.55 or 5.75.Feature supervision does not improve MAE-based tokens, so the authors do not pursue these configurations further.Source: §C.2 · p. 18

C.3 Reconstruction results

Feature supervision reduces pixel-level fidelity but generally improves generation-oriented evaluations; token-structure diagnostics show that neither reconstruction nor linear separability consistently explains generation quality.

C.3 Reconstruction results

  • Across four encoders, rFID ranges from 0.72 to 1.40; feature targets improve rFDr6 for SigLIP, SigLIP2, and DINOv3, but not DINOv2.The reported rFDr6 changes are 2.68 to 2.45, 2.09 to 2.04, and 2.24 to 2.08 for the first three encoders, versus 1.82 to 2.06 for DINOv2.Source: §C.3 · p. 18
  • Feature supervision lowers PSNR and SSIM for all four encoders, including DINOv3 PSNR from 20.61 to 17.45, while favoring generation-oriented distribution and reward-model evaluations.The authors note that feature-supervised tokens discard some input detail, which may not all be perceptually irrelevant.Source: §C.3 · pp. 18–19
  • Feature-only loss outperforms pixel and combined losses for three encoders, while feature-only supervision with joint global-and-patch modeling achieves the best gFID for all four.Table 11 reports rFID and unguided gFID, with lower values better.Source: Table 11 · p. 19

C.4 Token Structure Measurement

  • Feature targets raise adjacent-patch bit dependence for SigLIP, DINOv2, and DINOv3, but lower it for SigLIP2; the measure is a component-wise proxy, not full-token mutual information.Figure 7 pairs linear-probe accuracy on 25K test images with patch-only gFID on 50K unguided samples, with a log-scaled gFID axis.Source: §C.4 · p. 20, Figure 7 · p. 20
  • Linear-probe accuracy and gFID move inconsistently across encoders: both worsen for SigLIP2, both improve for DINOv3, and diverge for SigLIP and DINOv2.The authors conclude that linear separability does not consistently track class-conditional generation quality.Source: §C.4 · p. 20

D Evaluation protocol

The evaluation covers class-conditioned ImageNet generation at two resolutions and text-conditioned benchmarks, using distribution metrics and documented sampling configurations.

D Evaluation protocol

  • Evaluation covers ImageNet-1k class-conditioned generation at 256 × 256 and 512 × 512, plus text-conditioned generation on GenEval, GenAI-Bench, and DPG-Bench.Generation metrics use 50K samples, and PRDC compares them with 10K real reference images.Source: D Evaluation protocol · p. 20
  • gFDr6 averages normalized Fréchet distances across six feature spaces, while rFDr6 applies the same aggregation to reconstructions; gFID and rFID use Inception features.The paper reuses released reference statistics for these measurements.Source: D Evaluation protocol · p. 20
  • Reported comparisons distinguish author-evaluated results from transcribed baselines and exclude BAR’s additional 40-epoch decoder-only refinement from the primary comparison.Sampling configurations for FeaTok’s 256 × 256 results are listed in Table 13.Source: D Evaluation protocol · pp. 20–21, Table 13 · p. 21

E Sampling hyperparameters

  • FeaTok’s patch and global branches can use separate guidance scales, temperatures, and bit-group allocations; a k-group schedule requires k+1 forwards per position.The allocation length sets sampling cost, and scale 1.0 denotes no guidance.Source: E Sampling hyperparameters · p. 21

F Comparison using conventional gFID and PRDC metrics.

At low gFID values, improvements in Inception distance can conflict with fidelity diagnostics, motivating comparisons across multiple representations and complementary measures. Under gFID-selected configurations, most FeaTok models still improve on BAR in gFDr6.

F Comparison using conventional gFID and PRDC metrics.

  • gFDr6 averages normalized Fréchet distances across six representation spaces, whereas gFID uses Inception-v3 features and can misrepresent sample quality at low values.ImageNet validation images score 1.68 against the training set; the passage describes values below 2.0, especially below 1.68, as a regime where gFID no longer tracks sample quality reliably.Source: F Comparison using conventional gFID… · p. 22
  • At low-gFID operating points, guided settings lower gFID and raise recall across ten BAR and FeaTok models, but lower precision in seven and density in nine.The passage cautions that a small gFID improvement is not an unambiguous improvement in generation quality.Source: F Comparison using conventional gFID… · p. 22
  • For BAR-B, guidance changes gFID from 1.64 to 1.13, precision from 0.800 to 0.775, density from 1.153 to 1.060, and recall from 0.614 to 0.660.Coverage remains near saturation at 0.994–0.997.Source: F Comparison using conventional gFID… · p. 22
  • All four FeaTok-B models and three of four FeaTok-L models improve on same-size BAR in gFDr6 under gFID-selected configurations; FeaTok-L-DINOv3 falls from 4.57 to 3.35 unguided and 4.11 to 3.45 guided.FeaTok does not use BAR’s additional 40 epochs of decoder-only fine-tuning; the comparison also holds against BAR with the refined decoder in the reported DINOv3 settings.Source: F Comparison using conventional gFID… · p. 23, Table 15 · p. 23

G Sampling-cost measurement

Sampling cost includes both generation and pixel decoding, with throughput and memory measured under specified hardware and batch conditions. FeaTok-L reaches peak end-to-end throughput 8.1× and 4.2× RAEv2’s in unguided and guided settings, respectively.

G Sampling-cost measurement

  • TFLOPs count the full sampling loop plus one pixel-decoder pass in a hardware-independent 2×MAC convention.Classifier-free guidance roughly doubles generator cost for BAR and FeaTok, while the decoder is not duplicated; its 0.16-TFLOP cost adds 15–32% to their generator cost.Source: G Sampling-cost measurement · p. 23
  • Throughput is measured on one NVIDIA L40S GPU in bf16 without compilation, with generation-only and end-to-end rates timed separately.End-to-end throughput includes pixel decoding; peak memory is measured during sampling and decoding at the stated batch size, not per image.Source: G Sampling-cost measurement · p. 23, Table 16 · p. 24
  • FeaTok-L reaches 15.35 and 7.75 images/s at peak end-to-end throughput, 8.1× and 4.2× RAEv2’s corresponding peaks.Table 16 reports peak batch sizes and peak GPU memory alongside throughput; the measurements use one L40S GPU in bf16 without compilation.Source: Table 16 · p. 24, G Sampling-cost measurement · p. 24
  • Peak end-to-end throughput uses each model and guidance setting’s best batch size from a sweep of 1 to 2048.Autoregressive models peak at batches 256–1024; RAEv2 peaks at batch 8 on this GPU.Source: G Sampling-cost measurement · p. 24

H Qualitative samples across tokenizer encoders

The qualitative ImageNet panels compare released BAR-L samples with FeaTok-L samples using four pretrained encoders.

H Qualitative samples across tokenizer encoders

I Qualitative text-to-image samples

The text-to-image visuals show MJHQ-30K animal and food prompts at 256 × 256, with full numbered prompts provided separately.

I Qualitative text-to-image samples

Related papers

Loading 2610.22060v1…