On the Diffusibility of High-Dimensional Latents

1 Introduction

• Pretrained semantic encoders are not explicitly trained to reconstruct images, so they may lose high-frequency details such as text, fine textures, and small objects.Finetuning with an autoencoding loss is presented as a way to address this detail loss.Source: §1 · pp. 1–2
• Reconstruction-tuned feature spaces improve detail preservation, but standard velocity prediction converges slowly even in a simple single-image overfitting test.Prior compression-based approaches can discard important information, leaving generation directly in these high-dimensional spaces unexplored.Source: §1 · p. 2
• Although reconstruction-tuned representations have large feature dimensions, their features concentrate near a much lower-dimensional subspace, producing effective-dimensionality collapse.Effective dimensionality is the minimum number of components needed to explain a specified percentage of total variance.Source: §1 · p. 2
• In this geometry, the velocity target includes an orthogonal component that can dominate in high dimensions, making optimization inefficient and degrading sample quality.The paper proposes x0-prediction to focus learning on the low-dimensional signal and avoid explicit regression of orthogonal noise directions.Source: §1 · p. 3
• Across two strong-reconstruction tokenizers, x0-prediction consistently improves text-to-image performance on GenEval, DPG-Bench, and COCO-30k FID, matching or surpassing frozen semantic baselines.The paper reports this outcome for fine-tuned DINOv2-L and an MAE-based RAE.Source: §1 · p. 3

2.1 Diffusion and Flow Matching

• Flow matching trains a network to predict the velocity field along a straight-line path between noise and data; diffusion and flow matching differ in path and parameterization under a unified probability-path view.Diffusion models reverse a scheduler-defined path and may predict noise, clean data, velocity, or the score function.Source: §2.1 · pp. 3–4

2.2 Unifying Representation Learning and Flow Matching

• Prior representation-learning work either aligns diffusion-model features with pretrained representations or redesigns the latent space through representation autoencoders; this paper follows the latter direction.The paper extends AlignTok to high-dimensional spaces and argues that RAE-style designs alone do not fully address their diffusibility challenges.Source: §2.2 · p. 4

2.3 Clean image (x0) prediction

• Prior work revisits x0-prediction for high-dimensional pixel space, while this paper studies it in high-dimensional representation spaces and connects its benefit to reconstruction-induced effective-dimensionality collapse.Diffusion parameterizations include predicting reverse-process statistics, noise, clean data, velocity, or the score function.Source: §2.3 · pp. 4–5

3.1 Preliminaries

• Standard flow matching linearly interpolates clean data with isotropic Gaussian noise and uses their difference as the velocity target.The paper reviews this setup before analyzing its optimization in high-dimensional representation spaces.Source: §3.1 · p. 5, §3 · p. 5

3.2 Strong-Reconstruction Representation Autoencoder

• Effective dimensionality is assessed from centered, L2-normalized patch embeddings using singular values and the minimum component count explaining a chosen variance threshold.The analysis uses 200k ImageNet patches per model and evaluates thresholds of 90%, 95%, and 99%.Source: §3.2 · p. 6
• The encoder-decoder setup includes frozen encoders and a reconstruction-finetuned DINOv2-L encoder, with reconstruction PSNR and effective dimensionality evaluated on ImageNet validation data.The text-to-image framework uses high-dimensional visual features as latents and a pretrained autoregressive model with learnable query tokens to condition a randomly initialized DiT.Source: Table 1 · p. 6, §3.2 · pp. 7, 9
• In the assumed low-dimensional subspace, x0-prediction can model the underlying latent more efficiently when the ambient dimension is much larger.The analysis contrasts this with velocity prediction, which must allocate capacity to orthogonal noise directions.Source: §3.2 · p. 9
• The DiT is trained to predict clean image features from noisy representations, using text embeddings as conditioning.The loss compares the predicted clean feature with the target clean feature, scaled by 1/t.Source: §3.2 · p. 9

4.1 Implementation Details

• The experiments use fine-tuned DINOv2-L and MAE-RAE tokenizers, with the latter's encoder pretrained on ImageNet-1k and frozen during decoder training.Source: §4.1 · p. 11, §4.3 · p. 13

4.2 Evaluation

• Generation is evaluated with FID on COCO-30k and GenEval and DPG-Bench for text-image alignment, using a 100-step Euler sampler.Source: §4.2 · p. 11

4.3 Text-to-image Generation Comparison

• For fine-tuned DINOv2-L, x0 examples show better overall visual quality, including object structure and text alignment, than v-prediction examples.Source: Figure 5 · p. 12, §4.3 · p. 12
• Across fine-tuned DINOv2-L and MAE-RAE, x0 prediction improves generation over v prediction; for DINOv2-L it also surpasses the original tokenizer with v prediction.Source: Table 2 · p. 11, §4.3 · pp. 12–13, Table 3 · p. 13
• FID is 16.80 for high-dimensional fine-tuned DINOv2-L with x0 prediction, versus 17.64 for its 32-dimensional compressed variant with v prediction.The compressed variant has stronger GenEval and DPG-Bench text-alignment metrics, while the high-dimensional approach has better visual fidelity.Source: Table 2 · p. 11, §4.3 · p. 14
• The larger-dataset experiment trains on 90M examples, then applies supervised fine-tuning at 256 and 512 resolution; the authors describe the results as suggesting scalability potential.Source: §4.3 · pp. 14–15, Table 4 · p. 14

• Across two strong-reconstruction tokenizers, x0 prediction consistently improves text-to-image generation quality.Source: §5 · p. 15

• DINOv2-L tokenizer training combines semantic preservation, L1, perceptual, and adversarial losses.Source: §A.1 · p. 22
• The authors set ωsp = 1, ωp = 1, and ωg = 0.5, following [7].Source: §A.1 · p. 22
• The DiT has 24 layers, 24 attention heads per layer, a 1536-dimensional hidden size, and SwiGLU activation; its latent inputs are 1024 × 16 × 16 for fine-tuned DINOv2 and 768 × 16 × 16 for MAE.Source: §A.1 · p. 22

• Effective-dimensionality results on ImageNet-21k, BLIP3o, and CC12M are consistent with those in Table 1.Source: §A.2 · p. 22

A.3 Understanding Performance of Finetuned Tokenizer

• ImageNet-1k linear-probe accuracy is 84.5% for original DINOv2-L and 63.9% for the finetuned tokenizer.The passage uses linear probing to assess understanding capability.Source: §A.3 · pp. 22–23
• PSNR is 17.34 for original DINOv2 and 29.12 for finetuned DINOv2.Source: §A.3 · p. 23
• Preserving good understanding capability while improving reconstruction performance for one tokenizer is left for future research.Source: §A.3 · p. 23

A.4 Visual Understanding of Frozen VLM Backbone

• The frozen Qwen3-VL-2B-Instruct backbone is evaluated as a potential foundation for text-conditioned generation.Source: §A.4 · p. 23
• The evaluation covers seven benchmarks: MMMU, MMMU-Pro, MathVista, MathVerse, MathVision, MMStar, and Charxiv-RQ.Source: §A.4 · p. 23
• Table 7 reports the frozen backbone's performance across those visual reasoning and understanding benchmarks.Source: Table 7 · p. 23, §A.4 · p. 23

A.5 Extended Derivations for High-Dimensional Flow Matching

• The setup assumes x0 = Qc0, with orthonormal columns of Q; Gaussian noise then splits into independent parallel and orthogonal components.The noisy state is the sum of the subspace state and t times the orthogonal noise.Source: §A.5 · p. 24
• The orthogonal noise is recoverable deterministically from the high-dimensional noisy state for t > 0, while its projection onto the subspace yields the manifold state.The optimal high-dimensional velocity predictor combines the low-dimensional velocity predictor with the orthogonal term (1/t)P⊥z_t^h.Source: §A.5 · pp. 24–25
• The high-dimensional velocity target contains a deterministic orthogonal component that adds no information about the clean latent but still consumes model capacity.Its expected squared norm is h−l, whereas the low-dimensional velocity target has expected squared norm l+E[∥c0∥2].Source: §A.5 · p. 25
• For finetuned DINOv2-L, h = 1024 and R90 = 129, indicating that many ambient dimensions lie outside the effective signal subspace.Source: §A.5 · p. 25
• Under the subspace model, the Bayes-optimal x0-predictor outputs vectors in col(Q), with no orthogonal noise component in its clean-data target.The simplified predictor is Q times the conditional expectation of c0 given the low-dimensional noisy state.Source: §A.5 · p. 26