Image Tokenizers Should Be Trained Between Images

1 INTRODUCTION

• Generated codes can fall outside the decoder's training codes, and decoded line quality predicts generation quality better than reconstruction quality.Existing noise-based training improves robustness around training codes but does not target the space between them.Source: §1 · pp. 1–2
• InterpTok mixes two training-image codes and decodes them, using a discriminator for naturalness and a mixing-ratio loss to prevent copying either source.Together, the losses supervise plausible compositions at the correct ratio without target images between training samples.Source: §1 · p. 2
• Regularization losses align codes with pretrained vision-model features, replacing MAETok's auxiliary masked-feature decoders while retaining its reconstruction objective.The paper reports that omitting latent-structuring losses makes codes harder for the generator to learn.Source: §1 · p. 2
• InterpTok improves interpolation quality more than threefold and generation quality by more than a quarter while keeping MAETok's reconstruction quality.It also lets its generator train and sample faster.Source: §1 · p. 2
• For MAETok's already structured latent space, fine-tuning only the decoder can improve existing generators without retraining them.The paper describes this as a cheap way to adapt the decoder.Source: §1 · pp. 2–3

2 RELATED WORK

• Earlier tokenizers train decoders on single-image codes, while later approaches shape latent spaces through patch-independent or pretrained-feature-aligned codes.MAETok uses masked modeling and auxiliary decoders that predict features of hidden patches, then fine-tunes its pixel decoder for detail.Source: §2 · p. 3
• Generated codes can deviate from training codes, and decoded-mixture quality predicts generation quality better than reconstruction quality.Prior decoder training addresses unfamiliar codes through code transformations, noise mixing, or post-training on generator samples.Source: §2 · p. 3
• Earlier interpolation methods target small-image representation learning, while CompTok swaps real token subsets and does not train on the line it measures.InterpTok instead trains on continuous code mixtures and uses a mixing-ratio loss; the swapped-versus-generated gap applies to continuous, not discrete, tokens.Source: §2 · p. 3
• The overview shows two images encoded as a and b, their reconstructions, and decoded mixture ABα, with realism and ratio objectives applied to the mixture.The discriminator compares the decoded mixture beside a source, while the ratio loss keeps it α of the way between the sources.Source: Figure 2 · p. 4

3 METHOD

• InterpTok adds mixture decoding and semantic latent-space regularization to MAETok’s reconstruction objective.Its interpolation objective uses a discriminator and two mixing-ratio losses, while regularization structures the latent space.Source: §3 · p. 4

3.2 THE SPACE BETWEEN TWO CODES

• Diffusion generators tend to produce codes between training-image codes, but tokenizers are trained only on encoded training images, leaving intervening codes unsupervised.Random codes are a less targeted alternative because they mostly fall outside the training distribution; mixtures instead come with two sources and a ratio.Source: §3.1 · p. 4, §3.2 · p. 5

3.1 BACKGROUND

• The encoder maps images to compact codes and the decoder reconstructs them; MAETok uses masked-image feature prediction, then fine-tunes its decoder on complete-image codes.MAETok’s reconstruction objective combines pixel, perceptual, and adversarial losses.Source: §3.1 · pp. 4–5

3.3 INTERPOLATION OBJECTIVE

• The interpolation discriminator compares each decoded mixture with its sources, while mixing-ratio losses prevent source-copy solutions and encourage sharp compositions of both images.The discriminator judges whether mixtures look as real as their sources; pixel and feature losses enforce the mixing ratio.Source: §3.3 · p. 6

3.4 REGULARIZATION LOSSES

• Three feature-alignment losses and a representation Fréchet loss structure the latent space, aligning codes with semantic features and making similar images receive similar codes.The alignment losses also make image features predictable from codes and decoder activations; the Fréchet loss compares decoded-image and training-image feature distributions.Source: §3.4 · p. 6

3.5 FULL OBJECTIVE

• The full InterpTok objective combines reconstruction losses on single images, interpolation losses on mixtures, and regularization losses.The decoder is also given six appended blocks for capacity between images, though reported gains do not depend on them.Source: §3.5 · p. 7

4.1 SETUP

• Setup: InterpTok trains MAETok’s encoder and decoder on ImageNet at 256 × 256, replacing masking; evaluations include ImageNet and MS-COCO.Generation comparisons use class-conditional SiT-B models sampled without guidance.Source: §4.1 · p. 7, Table 2

4.2 MAIN RESULTS

• InterpTok cuts ImageNet iFID from 14.42 to 4.30, more than threefold and below every tokenizer in the comparison.The comparison covers nine of the thirteen tokenizers benchmarked by Xu et al. (2026).Source: Table 1 · p. 7, §4.2 · p. 8
• 25× versus 7×: at the midpoint, MAETok’s FID is 25 times its code-level FID, while InterpTok’s is 7 times; most gain arrives within 13,000 fine-tuning steps.Figure 3 tracks FID from a code to its midpoint and InterpTok’s iFID over fine-tuning.Source: §4.2 · p. 8, Figure 3 · p. 8
• 11.61 to 8.48: replacing MAETok with InterpTok lowers gFID; at one-eighth the sampling steps, its generator does even better while MAETok’s worsens.InterpTok’s generator also reaches MAETok’s final gFID in half the training steps.Source: §4.2 · p. 8, Table 2
• ImageNet rFID stays within 3% of MAETok’s, while PSNR, SSIM, and LPIPS improve on ImageNet and MS-COCO; MS-COCO rFID is higher.Source: §4.2 · p. 8
• InterpTok’s linear probe predicts class more accurately than MAETok’s, and 73% of its codes lie nearest their own class average, versus 55% for MAETok.The higher class discriminability is relevant to class-conditioned generation.Source: §4.2 · p. 8

4.3 ABLATIONS

• Interpolation alone improves interpolation quality but weakens probe accuracy; regularization alone improves probing but barely interpolation and yields the highest variant gFID.Combining both losses lowers gFID, consistent with the need for both learnability and decodability.Source: §4.3 · p. 9, Table 3 · p. 9
• 11.61 to 8.43: the decoder-only variant improves gFID on frozen MAETok codes; noise augmentation reaches 8.83 but barely changes interpolation and more than doubles rFID.Removing the feature mixing ratio loss raises iFID from 4.48 to 5.23 and gFID from 8.43 to 8.90; removing source conditioning raises iFID only to 4.69. With fewer sampling steps, only InterpTok’s generator improves gFID.Source: §4.3 · pp. 9–10, Table 3 · p. 9

• Diffusion generators tend to produce codes between training images, and the quality of decoded images there predicts generation quality.Source: §5 · p. 10
• InterpTok trains on mixtures of two training-image codes using an interpolation discriminator, a mixing ratio loss, and regularization losses that structure the latent space.Source: §5 · p. 10
• InterpTok improves interpolation quality more than threefold and lowers gFID by more than a quarter while keeping reconstruction quality.Its generator reaches MAETok’s final gFID in half the training steps and an even lower gFID with one-eighth the sampling steps; its decoder alone also improves existing MAETok generators.Source: §5 · p. 10
• The authors identify real-time animation, video generation, and granular editing as possible directions for future work.Source: §5 · p. 10
• The work evaluates on ImageNet and MS-COCO and collects no human-subject data.Source: ETHICS STATEMENT · p. 10

OVERVIEW

• The appendix covers generated samples and generation quality over training, code-space representations, reconstruction, multi-code mixtures, sampling-step effects, and implementation details.It also describes an earlier small-code-size experiment and its more semantic scene compositions.Source: OVERVIEW · p. 16

A ADDITIONAL VISUALIZATIONS

• Figures 4–16 show nine interpolation positions for 13 held-out ImageNet image pairs, comparing MAETok, SD-VAE, and InterpTok fine-tuned for 13k, 41k, or 82k steps.Figure 17 shows three-image mixtures: MAETok produces source overlays, whereas InterpTok produces a single mountain scene.Source: A ADDITIONAL VISUALIZATIONS · p. 16

B.1 GENERATION

• InterpTok yields lower gFID than MAETok at every measured generator budget, with the gap widest early and settling near a quarter of MAETok’s gFID.The matched MAETok control scores gFID 11.61, versus 13.73 for a published MAETok generator configuration; Table 4 also lists published configurations with differing generator sizes and budgets.Source: §B.1 · p. 21, Table 4 · p. 21

B.2 DECODING A FIXED GENERATOR’S SAMPLES

• For SiT-B samples on MAETok codes, InterpTok’s decoder lowers Fréchet distance in all four feature spaces, including DINOv2, which its losses do not use.For samples from a released SiT-XL, the MAETok decoder scores 2.46 and the InterpTok decoder 2.26; the generator and sampling setup are held fixed within comparisons.Source: §B.2 · pp. 21–22, Table 6 · p. 22
• The decoder’s gFID advantage widens as sampling steps are reduced, while Figure 18 tracks the gap from MAETok across generator training budgets.The figure marks the first budget at which InterpTok reaches MAETok’s gFID at 400,000 steps.Source: §B.2 · p. 22, Figure 18 · p. 22

B.3 IMAGE REPRESENTATIONS

• Under the same linear-probe procedure, SD-VAE codes predict classes less accurately than MAETok codes; the UMAP panels project 50,000 ImageNet validation codes per tokenizer, colored by class.RAE’s reported probe uses a different frozen-DINOv2 procedure, so it is not directly comparable.Source: §B.3 · p. 22, Figure 20 · p. 23
• Figure 19 presents selected unguided samples in three-row blocks: MAETok above InterpTok fine-tuned for 41k and 82k steps, with columns sharing class and initial noise.The caption specifies the sample arrangement rather than an aggregate quality result.Source: Figure 19 · p. 23

B.4 EVALUATION DURING TRAINING

• The random-pair midpoint FID falls more than sixfold by 41,000 steps and changes little afterward, while both the 41,000- and 82,000-step checkpoints yield lower gFID than MAETok.The 41,000-step checkpoint is reported because its generator reaches the lowest gFID; the matched generators are evaluated at 250 sampling steps without guidance.Source: §B.4 · pp. 23–24, Table 7 · p. 24, Table 8 · p. 24
• Reconstruction FID continues falling after 13,000 steps, and linear-probe accuracy continues improving across the fine-tune.Although the 13,000-step checkpoint has the lowest iFID, its generator has higher gFID than the generator on 41,000-step codes, so the authors interpret generation as depending on more than iFID-measured decodability.Source: §B.4 · pp. 23–24

B.5 RECONSTRUCTION ON IMAGENET AND MS-COCO

• InterpTok has higher interpolation quality than MAETok across every measured pairing, number of codes, and dataset, with its largest advantage near the line’s midpoint.On MS-COCO, where neither tokenizer is trained, its midpoint FID is six times lower than MAETok’s; among the compared tokenizers, SD-VAE has the lowest reconstruction FID but the highest midpoint FID.Source: §B.5 · p. 24, Table 9 · p. 24
• In DINOv2 features, InterpTok’s midpoint Fréchet distance is three times lower, and its 41,000-step checkpoint scores lower than the 82,000-step checkpoint in SigLIP2 and MAE features.The two checkpoints reach about the same midpoint distance in DINOv2; Table 10 compares decoded MS-COCO midpoints with COCO sources, with values comparable within columns.Source: §B.5 · p. 24, Table 10 · p. 24

B.6 MIXTURES OF SEVERAL IMAGE CODES

• For equal-weight mixtures across classes, InterpTok’s decoded two-code FID is six times lower than MAETok’s and remains slightly lower for ten-code averages.On same-class pairs, the Inception Score of InterpTok mixtures is about the same as that of its reconstructions.Source: §B.6 · p. 25, Table 12 · p. 25

B.7 DECODED PATHS

• InterpTok’s decoded paths have lower CMMD than MAETok, SD-VAE, and crossfades, and nearly all paths move in order from the first source toward the second.The evaluation distinguishes perceptual path length, largest-step share, midpoint distance from a source, and ordered movement across nine positions; path quality is compared with held-out real images.Source: §B.7 · p. 25, Table 14 · p. 26, Table 13 · p. 26

B.8 INTERPOLATION POSITION AND GEOMETRY

• InterpTok’s advantage over MAETok is largest near the midpoint, and its midpoint FID on a spherical path is even lower than on the straight path.The section notes that a straight line need not be the shortest path in a curved latent space; diffusion interpolation methods instead change the generator or sampling.Source: §B.8 · p. 25, Table 15 · p. 26

Fewer steps

• At 8 deterministic steps, gFID improves from 16.17 to 6.38 for InterpTok and from 13.25 to 7.50 for MAETok’s codes with InterpTok’s decoder, while MAETok worsens.At 250 steps, all tested models perform worse with the deterministic than the stochastic sampler; with fewer steps, the stochastic sampler’s gFID improves only for InterpTok, down to 32 steps.Source: §B.9 · pp. 27–28
• At 8 deterministic steps, codes whose nearest class average is their own rise from 71% to 87% for InterpTok and from 38% to 48% for MAETok.Generated codes of all models are 15–18% closer to their class average at 8 than at 250 steps; with the stochastic sampler, the distances shrink by 13% for InterpTok and 4% for MAETok.Source: §B.9 · p. 28, Table 17 · p. 28

Early and late steps

• After one step, predictions are 2.3 times closer to InterpTok’s class average than to another class average, versus 1.5 times for MAETok; by step 250, the gap narrows.The figure tracks normalized distances to the generator’s own-class average and the nearest other-class average across 250 deterministic steps.Source: §B.9 · p. 28, Table 18 · p. 29, Figure 21 · p. 29

Nearest codes

• With fewer steps, generated codes have more same-class nearest neighbors, and InterpTok consistently has about three times as many as MAETok.The authors interpret this as codes around InterpTok’s class average more often resembling mixtures of that class, while MAETok’s resemble mixtures including other classes.Source: §B.9 · pp. 29–30

The encoder

• With InterpTok’s encoder, 73% of validation codes are closer to their own-class average than other class averages, versus 55% with MAETok’s encoder.The authors attribute this encoder property mainly to regularization losses; codes without those losses fare worst on the reported class-average measures.Source: §B.9 · p. 30, Table 21 · p. 31

The decoder

• For averages of two validation codes, InterpTok’s decoder yields Inception Score 139 versus MAETok’s 48, and FID 3.79 versus 35.12.For averages of three and five codes, InterpTok’s decoder reaches Inception Scores 131 and 92, versus 25 and 11 for MAETok; the authors report that the decoder makes codes near class averages more natural.Source: Table 19 · p. 29, §B.9 · p. 31

Generated images

• With the deterministic sampler, the share of decoded images classified as their conditioning class rises from 61% to 83% for InterpTok between 250 and 8 steps, versus 62% to 71% for MAETok’s codes with InterpTok’s decoder.With the stochastic sampler, InterpTok’s share rises from 68% to 71% at 32 steps, while MAETok’s falls from 64% to 63%.Source: §B.9 · p. 31, Table 22 · p. 32

Summary

• The authors interpret the experiments as support for their hypothesis on these generators, while explicitly not claiming proof or generality to generative models overall.Their proposed account links larger steps, codes closer to class averages, class-specific neighborhoods, and InterpTok’s mixture-trained decoder.Source: §B.9 · pp. 27, 31, 32

B.10 A SECOND TRAINING SEED

• A second SiT-B/1D generator per tokenizer, trained with seed 1, reproduces the generation-quality gap reported for seed 0.The generators train for 400,000 steps; the table caption specifies 250 sampling steps without guidance and 50,000 samples per cell.Source: §B.10 · p. 32, Table 23 · p. 32

B.11 FURTHER DRAWS OF THE RELEASED SIT-XL

• At 13,000 fine-tuning steps, the InterpTok decoder without six appended blocks matches the version with them, while removing feature mixing ratio loss raises gFID to 2.50, above MAETok’s 2.44.Across two further released SiT-XL draws, InterpTok’s decoder gives 2.24 in both, compared with MAETok’s decoder results of 2.44 and 2.49.Source: §B.11 · p. 32

B.12 UNCURATED SAMPLES

• The uncurated-sample figure compares the first eight randomly drawn samples per class for SiT-B on MAETok’s codes and InterpTok’s codes.The figure uses 250 stochastic-sampler steps without guidance.Source: §B.12 · p. 32

C INTERPOLATION UNDER A SMALLER CODE

• The experiment uses a frozen REPA-E autoencoder with a transformer encoder and flow decoder, trained through 32 register tokens of 32 dimensions each.Only the encoder and decoder are trained, and the code is normalized together onto a sphere.Source: C INTERPOLATION UNDER A SMALLER CODE · p. 33
• The smaller code’s mixtures change scene content more and superimpose the two images less; a beach-and-cliff mixture, for example, becomes one coastline.The appendix depicts positions along lines between two codes and mixtures of four corner codes in squares.Source: Figure 23 · p. 33, C INTERPOLATION UNDER A SMALLER CODE · p. 34, Figure 24 · p. 34
• Random-code training alongside interpolated codes collapsed onto a single incoherent image in the smaller-code experiment.The authors describe decoding a meaningful image from an arbitrary code as more ambiguous than decoding one between two image codes.Source: C INTERPOLATION UNDER A SMALLER CODE · p. 34
• A smaller bottleneck can make codes more semantic at the cost of reconstruction, while it can be the better choice for semantic interpolation.Table 25 compares reconstruction with MAETok and InterpTok; Figures 23 and 24 show two-code paths and four-code mixtures.Source: Figure 23 · p. 33, C INTERPOLATION UNDER A SMALLER CODE · p. 34, Table 25 · p. 34

D.1 TRAINING DETAILS OF THE TOKENIZER

• InterpTok fine-tunes both encoder and decoder, unlike MAETok’s second stage, which freezes the encoder; reconstruction alone does not fix latent-space shape.A non-affine invertible latent map can preserve individual reconstructions while changing straight-line interpolations; the training configuration is summarized in Table 26.Source: §D.1 · pp. 34–35, Table 26 · p. 35

D.2 TRAINING OBJECTIVES

• Each micro-batch contains single images and mixtures, with every pair of its single images producing a mixture and 3.5 times as many mixtures as single images.The objective description defines reconstructions and mixture inputs alongside frozen vision-model CLS features.Source: §D.2 · p. 35

1. Reconstruction Lrec = 1

• The pixel mixing ratio loss constrains the decoded mixture’s ratio between source reconstructions, while the interpolation discriminator selects a natural image rather than a pixel crossfade.The source reconstructions are fixed targets for this loss; many images satisfy the ratio constraint, so the discriminator supplies the naturalness criterion.Source: §1 · p. 35

5. Feature mixing ratio loss Lfeat

• Feature mixing ratio loss places the decoded mixture’s vision features along the shorter great-circle arc between source features; representation Fréchet loss uses frozen feature spaces and ImageNet reference moments.The feature-ratio loss acts on a subset of mixtures; the Fréchet loss acts on decoded images, including mixtures, with features from three stated networks.Source: §5 · p. 36, §6 · pp. 36–37

D.3 DISCRIMINATOR ARCHITECTURE

• The interpolation discriminator compares each decoded mixture with each source in both directions, averaging four passes before the hinge loss.It uses five layers of a frozen DINO ViT-S/16 backbone; the separate reconstruction discriminator is unconditional to avoid classifying departures from the source as fake.Source: §D.3 · p. 37

D.4 DECODER ARCHITECTURE

• The decoder appends six gated blocks to MAETok’s 12-block decoder, with each gate initialized to zero.The added blocks contribute 43.1M parameters to the released checkpoint’s 173.9M.Source: §D.4 · p. 37

D.5 PAIR SAMPLING

• Training pairs each image with its most similar unpaired neighbor among its k nearest neighbors in frozen vision-model CLS features; k grows from 8 to 128 over epochs.Each micro-batch mixes all 28 pairs among its eight images, including pairs from different selected image pairs.Source: §D.5 · pp. 37–38

D.6 INTERPOLATION COEFFICIENTS

• Each of the 28 mixtures draws its mixing coefficient from Beta(2, 2), a distribution that favors the middle of the interpolation line.The appendix identifies the midpoint as the harder region for decoding.Source: §D.6 · p. 38

D.7 EVALUATION PROTOCOL

• Every iFID and ImageNet rFID in Tables 1 and 11 uses Xu et al. (2026)'s released evaluation code at defaults; daggered values follow their source protocols.Table 28 lists corpus, sample count, pairing, and reference; Table 30 identifies evaluators.Source: §D.7 · p. 38, Table 28 · p. 38, Table 30 · p. 39
• Table 29 reproduces released MAETok and DM-VAE iFID and rFID measurements within a few percent; these gaps are not variance intervals.Chen et al. (2025a) report rFID 0.48 for these weights under a protocol that names neither reference statistics nor toolkit.Source: §D.7 · p. 38, Table 29 · p. 38
• Table 30 records quoted tokenizers' supervision, corpus, code dimensions, and evaluators; measured comparisons hold corpus and evaluator fixed within each column.Table 1 includes thirteen tokenizers in fourteen rows, while Table 30 lists additional reconstruction-only tokenizers and a 64-channel VA-VAE.Source: §D.7 · pp. 38–39, Table 30 · p. 39
• Table 11 evaluates pixels on the same ImageNet images, following MAETok's SSIM convention and computing LPIPS on AlexNet.The released code maps images to [0, 1] before treating them as [−1, 1]; Table 31 lists other pixel conventions.Source: §D.7 · p. 39, Table 31 · p. 39
• The released iFID pairing decodes spherical-path midpoints between validation codes and one of their ten nearest training codes, while training mixes codes along Eq. 1's straight line.The pairing comparison fixes corpus, reference, ADM evaluator, and straight-line interpolation, varying only the pairing (Table 31).Source: §D.7 · p. 39, Table 31 · p. 39
• SiT-B settings are given in Table 27; repeated draws of one generator can differ by up to 0.3 gFID, and dashes denote unmeasured values.MAETok's authors report no SiT-B.Source: §D.7 · p. 39