PaperlayerView exact revision on arXiv

Source-linked AI summary

Purify Before You Align: Improving Representation Alignment for Diffusion Models

Yingheng Wang, Yaoqiang Li, Yaqin Wu, Junwen Bai, Jiatao Gu, Christopher De Sa, Volodymyr Kuleshov

arXiv:2610.20761v1cs.CVcs.LG

TL;DR

Representation alignment leaves open which information in pretrained features is useful for diffusion generation. The paper introduces local appearance purification (LAP), aligning to residual features after removing locally predictable content. LAP improves generation across five visual teachers, and a constant-readout score predicts purification benefit across 13 encoders.

  • Problem

    Standard representation alignment uses pretrained features without distinguishing which target information is more useful for generation, and encoder choice remains largely empirical.

    Source: §1 · p. 2

  • Method

    LAP removes target content predictable from local appearance and aligns to the residual, while an information-theoretic analysis and constant-readout test characterize alignment targets.

    Source: §1 · p. 2

  • Results

    LAP improves generation across five visual teachers, with the largest gain for MAE; on ImageNet-256 with SiT-B/2, MAE-based FID falls from 53.1 to 42.3.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The constant-readout score predicts the benefit of purification across 13 encoders (Spearman ρ = 0.94).

    Source: §1 · pp. 2–3

    Evaluation centers on SiT and ImageNet; larger-model checks use two seeds, cross-domain runs are short, and gains depend on the teacher and recipe.

    Source: §6 · p. 10

Abstract

from arXiv · show

Representation alignment improves and accelerates diffusion training by encouraging intermediate features to match a pretrained visual encoder. However, standard alignment matches pretrained representations without distinguishing what information may be more useful for generation. We introduce local appearance purification (LAP), which improves generation quality and makes encoders such as MAE more effective alignment targets by removing locally predictable content and aligning to the residual. LAP modifies only the alignment target, and requires no changes to diffusion training or inference. Through controlled interventions within a fixed encoder, we show that effective alignment depends on what information the target emphasizes, not simply how much image information it contains. Our theory gives a variational interpretation of standard alignment and, for fixed local predictors, a sum of token-wise information bounds conditioned on local appearance. To understand why purification benefits some encoders more than others, we derive an exact constant-readout test that measures how well the target can be matched without using the diffusion model's hidden state. LAP improves generation quality across five pretrained representations (MAE, AIMv2, CLIP, DINOv2, DINOv3), with the largest gain for MAE: on ImageNet-256 with SiT-B/2, it reduces FID from $53.1$ to $42.3$. Its gains persist with stronger alignment recipes and tokenizers. Computed before diffusion training, the constant-readout score predicts the benefit of purification (Spearman $\rho=0.94$ across 13 encoders).

1 INTRODUCTION

The paper asks what information an alignment target should emphasize, since pretrained encoders produce different generation quality and reconstructive features do not necessarily yield stronger generation gains. Fixed-encoder interventions motivate local appearance purification (LAP), which improves generation across visual teachers and is especially beneficial for MAE.

1 INTRODUCTION

  • Standard alignment leaves the choice of encoder largely empirical, and different pretrained encoders produce very different generation quality.MAE predicts clean-image content better than DINOv2 yet yields smaller generation gains in the development comparison.Source: §1 · p. 2
  • Effective alignment depends on what information the target emphasizes, not simply how much image information it contains.With DINOv2 fixed, aligning to its locally predictable component performs worse than no alignment, while the residual retains the benefit; making the target more predictive of the denoising target also improves prediction but worsens generation.Source: §1 · p. 2
  • LAP improves generation across five visual teachers, with the largest gains for MAE; on ImageNet-256 with SiT-B/2, MAE-based FID falls from 53.1 to 42.3.Its gains persist with stronger alignment recipes and tokenizers, without changing the encoder or adding inference overhead.Source: §1 · pp. 2–3
  • A constant-readout test measures alignment achievable without the hidden state, and its score predicts purification benefit across 13 encoders (Spearman ρ = 0.94).The paper also gives an information-theoretic view of raw and purified alignment.Source: §1 · pp. 2–3

2 BACKGROUND

The background describes diffusion alignment to frozen visual-encoder features and introduces diagnostics that distinguish information available in representations from information the denoiser uses.

2.1 REPRESENTATION ALIGNMENT IN DIFFUSION TRAINING

  • REPA adds token-wise matching between projected denoiser hidden states and frozen visual-encoder features of the clean image to latent flow-matching training.The encoder output is the raw target, and purified targets occupy the same alignment slot.Source: §2.1 · p. 3

2.2 MEASURING REPRESENTATION CONTENT AND USE

  • Linear probes measure decodable representation content, while hidden-state subspace ablations test whether the denoiser relies on the aligned subspace.The probes quantify clean-image prediction utility, teacher-feature presence in the hidden state, and aligned-representation prediction of the velocity target; ablation effects are compared with rank-matched random subspaces.Source: §2.2 · p. 3

3 WHY PURIFY THE ALIGNMENT TARGET?

The section tests whether reconstruction-predictive information drives alignment benefits. Development comparisons and fixed-encoder interventions show that better prediction or locally predictable target content need not improve generation.

3 WHY PURIFY THE ALIGNMENT TARGET?

  • Better clean-image prediction generally accompanies worse generation across MAE, MoCo v3, DINOv2, I-JEPA, and CLIP.The comparison motivates fixed-encoder interventions because architectures, data, and pretraining objectives vary across encoders.Source: §3 · pp. 3–4, Figure 3 · p. 4

3.1 MORE VELOCITY-PREDICTIVE ALIGNED REPRESENTATIONS CAN HURT GENERATION

  • Velocity-prediction R2 rises from 0.11 to 0.53, while FID-50K worsens from 41.75 to 46.79 under an auxiliary velocity-prediction loss.The aligned subspace is decodable from the hidden state, but its ablation has a smaller velocity-error effect than rank-matched random subspaces; this tests that subspace, not all reconstruction-related information.Source: §3.1 · p. 4

3.2 LOCALLY PREDICTABLE CONTENT DOES NOT DRIVE THE ALIGNMENT BENEFIT

  • With DINOv2 fixed, aligning only to locally predictable content worsens FID from 53.92 without alignment to 56.71, while the residual reaches 41.47 versus 41.75 for raw DINOv2.The predictable component has rank at most 17 and about 2% of centered variance; controls argue that rank reduction or batch refitting alone does not explain the result.Source: §3.2 · p. 4

4 LOCAL APPEARANCE PURIFICATION FOR REPRESENTATION ALIGNMENT

LAP removes target content predictable from local clean-image appearance and aligns the denoiser to the residual. The section also interprets alignment through information bounds and introduces a constant-readout baseline for target structure shared across tokens.

4 LOCAL APPEARANCE PURIFICATION FOR REPRESENTATION ALIGNMENT

  • LAP modifies the alignment target by subtracting locally predicted content, leaving diffusion training and inference otherwise unchanged.The predictor selects what to remove and receives no gradients from diffusion training.Source: §4 · p. 4, Figure 4 · p. 5, §4.1 · p. 5

4.1 LOCAL APPEARANCE PURIFICATION

  • LAP-L subtracts an affine prediction from each clean-latent patch, whereas LAP-N uses a frozen convolutional predictor trained on local neighborhoods before diffusion training.The residual is not assumed to be purely semantic: local patches can contain semantic information, and a limited predictor may miss predictable content.Source: §4.1 · p. 5

4.2 A VARIATIONAL VIEW OF PURIFIED ALIGNMENT

  • Cosine alignment can be interpreted as probabilistic prediction of teacher-token directions, and the variational view bounds information in the hidden state about the target.For a fixed local predictor, purified alignment bounds a sum of token-wise information quantities conditioned on each token’s local appearance; the sum can count shared information more than once and is not joint mutual information.Source: §4.2 · pp. 5–6
  • The variational bounds may be loose and do not by themselves establish information separation or explain generation gains.The bound constants are independent of the denoiser and projector, and the smoothing scale is used only for the analysis.Source: §4.2 · p. 6

4.3 THE CONSTANT-READOUT TEST

  • The constant-readout test measures cosine alignment available from shared target structure without using the hidden state.Its target-only score can be computed before diffusion training by normalizing target tokens, averaging across images and positions, and taking the norm; comparing the trained readout score with this baseline measures the gain beyond it.Source: §4.3 · p. 6

5 EXPERIMENTS

Experiments evaluate LAP across teachers, alignment recipes, tokenizers, training settings, datasets, and the predictive value of C0. LAP most strongly helps weak targets such as MAE, while C0 is associated with LAP-L benefit.

5.1 EXPERIMENTAL SETUP

  • The default evaluation uses class-conditional SiT-B/2 on ImageNet-1k at 256 × 256 for 400K steps, with results averaged over two seeds.It reports FID-50K, Inception Score, precision, and recall; development experiments use dev-200 and FID-DINOv2 on 10K samples, which are not directly comparable to FID-50K.Source: §5.1 · p. 7

5.2 GENERATION QUALITY ACROSS TEACHERS

  • LAP-N reduces MAE FID from 53.05 to 42.30, approaching DINOv2-based REPA at 41.75.Across five teachers, mean FID improvements with LAP-L are largest for MAE (7.9), followed by AIMv2 (3.7) and CLIP (2.0); DINOv3 and DINOv2 gains are within seed variation.Source: §5.2 · p. 7, Table 1 · p. 7

5.3 LAP COMPLEMENTS DIFFERENT REPA-STYLE RECIPES AND TOKENIZERS

  • LAP improves MAE across stronger REPA-family recipes, including iREPA, VA-REPA, REG, and sREPA.For DINOv2, selected LAP variants reduce mean FID across all four recipes by 0.3–1.2, with lower FID in both seeds and higher mean IS and precision.Source: §5.3 · p. 8, Table 2 · p. 8
  • With frozen tokenizers, LAP-N reduces MAE FID from 53.05 to 42.30 with SD-VAE, 52.41 to 40.01 with EQ-VAE, and 33.74 to 27.56 with REPA-E VAE.Raw MAE is worse than no alignment with REPA-E VAE, whereas LAP also improves IS from 39.32 to 47.43 and precision from 0.574 to 0.606.Source: §5.3 · p. 8, Table 3 · p. 8

5.4 LAP GAINS PERSIST ACROSS TRAINING, SCALE, AND DATASETS

  • By 100K steps, LAP lowers FID-10K by 14.4 for MAE, 4.3 for AIMv2, and 4.5 for CLIP, with gains persisting through 400K steps.The trajectories use two-seed FID-10K means and ranges for SiT-B/2; LAP also remains effective at larger batch size, with SiT-L/2 and SiT-XL/2, and at a different alignment depth.Source: Figure 5 · p. 9, §5.4 · p. 9
  • On RESISC45 and AID, LAP with MAE achieves the best FID, outperforming DINOv2-based REPA by 4.9 and 3.6, respectively.It improves over no alignment by 6.6 FID on RESISC45 and 2.3 on AID; on AID, neither raw MAE nor DINOv2 improves over no alignment.Source: §5.4 · p. 9, Table 4 · p. 9

5.5 THE CONSTANT-READOUT SCORE PREDICTS PURIFICATION BENEFIT

  • Across 13 encoders, raw-target C0 correlates with LAP-L FID gains at ρ = 0.94 at 100K steps and 0.79 at 400K steps.The association remains after controlling for raw-target FID; C0 is a diagnostic for linear-purification benefit under this recipe, but does not explain the full benefit of LAP-N.Source: §5.5 · pp. 9–10, Figure 6 · p. 9

6 CONCLUSION

The conclusion presents LAP as a target transformation that makes weak alignment targets more useful and complements existing recipes without changing inference. The paper also identifies limits in evaluation scope and in what LAP and C0 establish.

6 CONCLUSION

  • LAP removes locally predictable content before alignment, and the conclusion reports improved generation for otherwise weak targets while complementing existing recipes and tokenizers.The paper also presents variational information bounds and C0, a score measuring alignment attainable without consulting the hidden state.Source: §6 · p. 10
  • Evaluation centers on SiT and ImageNet; larger-model checks use two seeds, cross-domain runs are short, and gains depend on the teacher and recipe.LAP does not guarantee a semantic residual, and C0 does not provide calibrated gain predictions.Source: §6 · p. 10

APPENDIX CONTENTS

A LINEAR-GAUSSIAN ANALYSIS OF DECODABILITY AND USE

This section develops theoretical analyses of decodability, predictive usefulness, and alignment, while limiting what they establish about diffusion training. It also proves an entropy lower bound without assuming finite entropy in advance and bounds perturbation effects on normalization and cosine similarity.

A.1 MODEL AND PREDICTION ERROR

  • The linear-Gaussian construction fixes h = ξ + η, with alignment target y = Cξ and prediction target v = Bξ; it yields optimal linear prediction errors for projected hidden states.The hidden state is fixed rather than learned, and the model assumes independent isotropic Gaussian observation noise.Source: §A.1 · p. 17

A.2 DECODABILITY CAN BE HIGH WHEN USE IS ZERO

  • When rank(C) < k, the target can be highly decodable as noise vanishes while its subspace has zero use; full-column-rank C instead gives normalized use one.The presence score depends on observation noise, not on B.Source: §A.2 · p. 17

A.3 PREDICTION THROUGH A RESTRICTED SUBSPACE

  • A direction improves prediction exactly when Bu ≠ 0, and a subspace preserves the full-state benefit exactly when B(I − P) = 0.The relevant overlap can be non-orthogonal, even when two row spaces have no literal intersection.Source: §A.3 · p. 18

A.4 SCOPE AND NUMERICAL VERIFICATION

  • The Gaussian model has no learning dynamics, so its propositions do not establish gradient competition, semantic content, or residual-alignment safety during training.The experimental intervention score need not be nonnegative or bounded by the full-layer effect, and the model’s optimal-predictor equivalence need not hold for a nonlinear denoiser.Source: §A.4 · p. 18

B.2 FROM THE VMF DECODER TO THE ALIGNMENT LOSS

  • Cosine alignment is a fixed-concentration vMF decoder’s expected log-likelihood, and minimizing it raises a lower bound on target information.The derivation applies to smoothed targets; the actual unsmoothed training objective is handled with a model-independent smoothing correction.Source: B ALIGNMENT TARGETS AS VARIATIONAL… · p. 18, §B.2 · pp. 19–20

B.1 SETUP AND NOTATION

  • For fixed local predictors, purified-target analysis sums token-wise information terms conditioned on each token’s local neighborhood, not the full clean latent.The sum may count shared information repeatedly, so it is not joint mutual information about the target array and depends on the chosen neighborhoods.Source: §B.1 · p. 19

B.2 FROM THE VMF DECODER TO THE ALIGNMENT LOSS

  • A product vMF decoder models token directions, making cosine alignment equivalent to maximizing expected log-likelihood without assuming independent target tokens.Factorizing the decoder is a modeling choice, not a claim that the true target tokens are independent.Source: §B.2 · pp. 19–20

B.3 GENERAL LEMMAS

  • Conditional Barber–Agakov bounds support the information analysis, and Gaussian smoothing gives directional targets densities with finite conditional entropy under the stated moment condition.The training targets themselves remain unsmoothed.Source: §B.1 · p. 19, §B.3 · p. 21
  • Jensen’s inequality and integrability controls establish the lower bound in Equation (30) without presupposing finite entropy.The proof bounds the negative part before negating the inequality.Source: §B.3 · p. 22
  • For nonzero vectors x and y, ||x̂−ŷ|| ≤ 2||x−y||/||y||, and cosine-similarity change is bounded by ||x̂−ŷ|| and by 2.The cosine bound follows from Cauchy–Schwarz, while the unit-vector distance is at most 2.Source: §B.3 · p. 22

B.4 RAW AND PURIFIED TARGETS

This section examines how purification changes the information denoiser features preserve and match, interpreting alignment losses as lower-bound objectives for raw or locally conditioned targets. It also clarifies the limits of these information bounds and the associated diagnostic protocols.

B.4 RAW AND PURIFIED TARGETS

  • Minimizing alignment loss maximizes its information lower bound, but does not guarantee that the bound or mutual information rises during training.Because constants differ across targets, alignment losses alone cannot rank their mutual information.Source: §B.4 · p. 24
  • As smoothing noise tends to zero, singular raw and conditional residual target laws drive the entropy terms to negative infinity, making the bounds vacuous even as the loss correction vanishes.The stated bounds require positive smoothing noise under their assumptions.Source: §B.4 · pp. 24–25
  • Minimizing residual regression error maximizes a lower bound on summed token-wise conditional information, with each decoder given its local prediction as side information.Overlapping neighborhoods can duplicate information across terms, so this sum is not a decomposition of the raw joint information quantity.Source: §B.4 · pp. 24–26

B.5 AN EXACT IDENTITY FOR USABLE INFORMATION

  • Usable information equals the best projector’s gain over the optimal shared constant-readout baseline and is nonnegative when the projector can realize every constant direction.The exact identity requires no density assumption or smoothing.Source: §B.5 · pp. 26–27

B.6 SCOPE

  • Because the hidden state is a deterministic function of the noisy input, its Shannon information cannot exceed the input’s and is unchanged by bijective reparameterizations.Training changes the current projector’s score, while the best score over its family need not be attained by a trained projector; neither is an identity for Shannon information.Source: §B.6 · p. 28
  • The information bounds clarify alignment’s objective but do not show that residuals are conditionally independent of local inputs or that predictable content harms generation.LAP’s selectivity depends on its restricted predictor class and must be assessed empirically; unlike InfoDiffusion’s learned auxiliary latent, alignment uses a frozen teacher target to shape denoiser features during training.Source: §B.6 · p. 28

C.1 TRAINING AND EVALUATION SETTINGS

  • The experimental protocol uses SiT-B/2 with alignment after block 8 and compares pretrained teacher representations across development and full-scale ImageNet-1k runs.The appendix states that diagnostics generally use seed 0 of a 400K-step ImageNet-1k checkpoint; generation results use the seed coverage stated in each table.Source: C EXPERIMENTAL AND DIAGNOSTIC PROTOCOLS · p. 29, §C.1 · pp. 29–30, Table 5 · p. 29

C.2 LINEAR PROBES AND SUBSPACE INTERVENTIONS

  • The diagnostics distinguish target presence, clean-latent prediction utility, and velocity sufficiency, and use image-disjoint evaluation for the main probes.Subspace ablations intervene on one layer rather than measuring every pathway through which alignment affects training; downstream computation may compensate for a local ablation.Source: §C.2 · pp. 30–31

C.3 IMAGE-DISJOINT RESULTS AND SPLIT SENSITIVITY

  • LAP-N (MAE) closes 62.8% of the unaligned-to-REPA presence gap, while auxiliary-loss velocity R2 rises from 0.115/0.112 to 0.534/0.533 across seeds 0/1.Aligned-subspace use rises from 12.2/10.6% to 18.9/11.5%, with a smaller change in seed 1; DINOv2 purification retains presence while reducing velocity prediction and normalized use. The random-subspace reference for seed-0 REPA averages 22.7% (95% interval: 21.7–23.7%).Source: §C.3 · p. 31, Table 6 · p. 31, Figure 7 · p. 32

C.4 REPRESENTATION CHANGES AFTER PURIFICATION

  • Purified MAE alignment raises held-out DINOv2 decodability from 0.130 to 0.178, closing about 63% of the gap to explicit DINOv2 alignment at 0.207.Purifying DINOv2 preserves teacher decodability while lowering linear velocity prediction and denoiser reliance on the aligned subspace; generation quality is preserved as this coupling decreases.Source: §C.4 · p. 32, Figure 7 · p. 32

C.5 DIAGNOSTICS ACROSS NOISE LEVELS

  • Across noise levels, REPA’s aligned-subspace use is lowest at high noise; predictable-only targets become more velocity-predictive toward low noise, while purified targets show the opposite trend.These observations use the stated probes and do not establish that the aligned subspace is the only route by which target information affects the denoiser.Source: §C.5 · p. 32, Table 7 · p. 33

C.6 REFERENCE SUBSPACES, RESAMPLING, AND RANK

  • At rank 32, LAP-N (MAE) has the largest aligned-subspace effect among MAE targets, while the decreasing DINOv2 sequence holds at ranks 32, 128, and 256.REPA stays close to the unaligned reference at ranks 32 and 128 and falls below it at rank 256. Resampling yields smaller normalized effects for REPA and purified targets, but changes the ordering of raw MAE and predictable-only models.Source: §C.6 · pp. 33–34, Table 10 · p. 34, Table 9 · p. 34

C.7 COMPONENT READOUT AND EASY-TO-MATCH CONTENT

  • The image-predictable component has higher presence than the residual (0.256 vs. 0.188), but the projector has higher cosine similarity to the residual (0.581 vs. 0.313).Random-token and dense token-split protocols retain this ordering, although token-held-out scores are higher; these measurements do not prove causal selection of semantic content because component geometry and scale can affect cosine.Source: §C.7 · pp. 34–35

C.8 CONSTANT-READOUT EVALUATION

The appendix tests constant readouts, purification variants, predictive information, seed uncertainty, and controls for target content. It finds that target structure and evaluation choices qualify how alignment scores and generation comparisons should be interpreted.

C.8 CONSTANT-READOUT EVALUATION

  • Raw MAE’s shared constant-readout baseline is 0.933 on dev-200 and 0.929 on ImageNet-1k, indicating high cosine without image-specific computation.The baseline is the norm of the sample mean of unit-normalized tokens; the logged training cosine of 0.957 uses different samples.Source: §C.8 · p. 35, Table 11 · p. 35
  • Position-only readouts still attain about 0.33 on LAP-N MAE, 0.21 on LAP-N DINOv2, and 0.10 on LAP-N CLIP, because purification does not explicitly remove position-specific means.For raw DINOv2 on ImageNet-1k, the shared-direction and per-position baselines are 0.16 and 0.31, respectively.Source: §C.8 · p. 35

C.9 POSITION-DEPENDENT PURIFICATION

  • LAP-P alone reaches 1499.4 on development-scale MAE, while LAP-N+P reaches 1453.8, 30.6 below LAP-N; the four purified DINOv2 targets lie within 15 points.These are development-scale values; the additional full-scale benefit of centering on top of LAP-N is not resolved by two seeds.Source: §C.9 · p. 36
  • At full scale, LAP-P reaches 44.32 FID-50K versus LAP-L’s 45.15, while LAP-N+P’s 42.07 versus LAP-N’s 42.30 difference is unresolved by two seeds.Position-dependent centering improves on LAP-L in this comparison, but its additional benefit on LAP-N is not resolved.Source: §C.9 · p. 36, Table 13 · p. 36

C.10 CLEAN-LATENT PREDICTION ACROSS NOISE LEVELS

  • MAE adds more linear clean-latent reconstruction information than DINOv2 or CLIP, including at high noise, although this does not yield the largest alignment benefit.At low noise, unaligned and REPA hidden states already predict clean-latent patches with R2 ≈0.95.Source: §C.10 · p. 37, Table 14 · p. 37

C.11 SEED COVERAGE AND UNCERTAINTY

  • With only two runs per configuration, the authors avoid precise significance or equivalence claims and report individual values rather than ranking close methods.Observed two-seed ranges are not confidence intervals.Source: §C.11 · p. 37

D.1 AUXILIARY PREDICTION CONTROLS

  • 1242.0/1252.3 versus REPA’s 1157.3/1161.2 at dev-200/100K, and 879.5/877.1 versus 825.1/821.1 at full scale, for auxiliary prediction.Standard FID-50K also increases from 41.75 to 46.79 at seed means; the results are consistent with interference through shared parameters but do not isolate a gradient-level mechanism.Source: §D.1 · p. 38

D.2 THE AFFINE TARGET DECOMPOSITION

  • Batchwise affine decomposition leaves approximately 2.7% residual norm on dev-200 and 2.1% at full scale, while target-direction cosine across recompositions is about 0.85 and 0.76.The measured variation quantifies changes from refitting each batch; it does not establish that they are negligible.Source: §D.2 · p. 38

D.3 SEPARATING RANK, INSTABILITY, AND PREDICTABLE CONTENT

Controls separate rank, target instability, and locally predictable content as explanations for alignment outcomes. Full-scale evaluations show predictable-only targets can fall below unaligned baselines.

D.3 SEPARATING RANK, INSTABILITY, AND PREDICTABLE CONTENT

  • Predictable content is more decodable, while the projector matches the residual more closely.The comparison uses one 400K REPA checkpoint, mid-noise, and disjoint fit/evaluation images.Source: Figure 8 · p. 39
  • A fixed random rank-17 DINOv2 projection scores 1521.7, close to the REPA seed range, showing that a rank-matched projection can remain effective.Because it projects image features, this control retains image-dependent information.Source: §D.3 · p. 39
  • Redrawing the projection each batch scores 1669.4, while frozen and nonlinear reconstruction targets score 1660.2 and 1677.0, respectively.Reconstruction-based targets remain ineffective without per-batch coefficient fitting; these controls weaken explanations based solely on rank or refitting noise.Source: §D.3 · p. 39
  • At full scale, the redrawn random control scores 937.9, close to unaligned scores of 945.7/947.4, and does not show the predictable-only target’s worse-than-unaligned endpoint.Its 100K–400K trajectory is 1273.8/1099.2/995.3/937.9.Source: §D.3 · p. 39

D.4 WHERE THE PREDICTABLE-ONLY TARGET BECOMES HARMFUL

  • At full scale, predictable-only targeting yields FID-50K 56.71 versus 53.92 for unaligned models, with scores of 1000.6/995.2 versus 945.7/947.4.The harm claim is limited to full-scale evaluations; development controls show loss of alignment benefit, not harm in every setting.Source: §D.4 · p. 39

E PURIFIER TRAINING, VALIDATION, AND ABLATIONS

  • LAP-L uses a batchwise affine fit, while LAP-N uses a frozen nonlinear predictor; both subtract local predictions but differ in fitting protocol and predictor inputs.The appendix also describes LAP-N validation, selection, and interactions with other alignment transformations.Source: E PURIFIER TRAINING, VALIDATION, AND… · p. 39

E.1 ARCHITECTURE AND VALIDATION CRITERIA

These appendices validate the purification setup, visualize how LAP changes MAE target geometry, and test receptive-field choices and interactions with spatial operations. They also check whether relational-structure changes track generation effects.

E.1 ARCHITECTURE AND VALIDATION CRITERIA

  • All listed predictors pass the class-probe feasibility checks; at full ImageNet-1k scale, MAE has the largest locally predictable fraction at both scales.The selection tolerance is a heuristic, not a significance test, and the probes do not certify complete semantic preservation.Source: §E.1 · p. 40

E.2 VISUALIZING THE MAE TARGET TRANSFORMATION

  • 0.876 raw-MAE mean pairwise cosine falls to 0.084 with LAP-L and 0.072 with LAP-N; these geometry scores alone do not identify which target best supports generation.The iREPA-style spatially normalized target scores −0.002, an empirical observation that does not by itself establish a generation advantage.Source: §E.2 · p. 42

E.3 RECEPTIVE-FIELD SELECTION AND THE CLIP CHECK

  • Three purified CLIP means lie within 6.5 points of one another and each improves on raw CLIP by about 29–35 points, but two seeds do not resolve the best transformation.CLIP’s smaller predictable fraction than DINOv2 accompanies a slightly larger observed development-scale gain, so ρ is not a universal ranking rule.Source: §E.3 · p. 43, Table 18 · p. 43

E.4 INTERACTION WITH IREPA-STYLE OPERATIONS

  • Combined convolutional-head and spatial-normalization operations improve raw MAE by about 140 points but increase LAP-N MAE’s mean by about 13 points, suggesting overlapping benefits without proving a common mechanism.At full scale, adding the operations to LAP-N (MAE) improves mean FID-50K from 42.30 to 41.32, so the interaction depends on training setting.Source: §E.4 · p. 44

E.5 A CHECK ON RELATIONAL STRUCTURE

  • MAE shows the larger relational-structure change while benefiting more from purification, so structural change does not track generation harm across the tested targets.The diagnostic does not rule out a role for spatial structure because it measures a different representation aspect than the intervention.Source: §E.5 · pp. 44–45

F ROBUSTNESS TO BATCH SIZE, CAPACITY, AND GUIDANCE

Robustness tests extend LAP across batch sizes, model scales, alignment depths, and guided sampling. LAP-N improves MAE in these settings, while the comparisons retain metric- and recipe-specific gaps to REPA.

F.1 LARGER BATCH AND MODEL

  • LAP-N remains close to REPA at SiT-B/2 batch 256, with FID-50K 28.57 versus 27.67, while the base-setting raw-MAE result should not be generalized to every recipe.The base result uses SiT-B/2, batch 64, and 400K steps; at SiT-L/2, LAP-N improves MAE at both tested batch sizes but remains behind REPA.Source: §F.1 · p. 45, F ROBUSTNESS TO BATCH SIZE, CAPACITY,… · p. 45
  • At SiT-XL/2 with batch 256, LAP-N (MAE) reaches FID-50K 9.67 versus REPA’s 8.10, improving raw MAE by 4.0 FID and recovering about 84% of the REPA gain.At SiT-L/2, LAP-N recovers about 76% and 81% of standard-FID gain at batch 64 and 256; REPA has higher Inception Score at both batch sizes.Source: §F.1 · pp. 45–46

F.2 ALIGNMENT DEPTH

  • At block 4, LAP gains persist and grow for MAE, AIMv2, and CLIP, while changes for DINOv2, DINOv3, and REG are small.Block 4 improves every raw target’s 400K FID-50K in the reported comparison.Source: §F.2 · p. 46, Table 21 · p. 47

F.3 CLASSIFIER-FREE GUIDANCE

  • Under CFG scale 1.5, MAE, CLIP, and AIMv2 gains persist in both seeds; LAP-N recovers 99% of REPA’s gain relative to the unaligned model.For DINOv2, DINOv3, and REG, guided differences change sign between seeds and means lie within 0.2 FID of one another.Source: §F.3 · p. 47, Table 22 · p. 48

G AN ENCODER-FREE ALIGNMENT CONTROL

  • The encoder-free control compares an EMA self-teacher’s raw target with a residualized target, aligning the student’s block-8 readout to the teacher’s layer-10 state.The teacher processes an independently noised copy of the same image; both variants standardize each feature dimension across batch tokens.Source: G AN ENCODER-FREE ALIGNMENT CONTROL · p. 48

G.1 CONSTRUCTION AND EVALUATION

The encoder-free control tests residualization while holding the self-teacher and training recipe otherwise identical. Under that tested recipe, residualization has no consistent advantage, and the comparison does not establish that pretrained encoders are necessary.

G.1 CONSTRUCTION AND EVALUATION

  • The student’s block-8 readout is aligned to an EMA teacher’s layer-10 hidden state, with the teacher processing an independently noised copy at t = 0.25.ARC removes the teacher-state component predicted by local clean and noised patches using an affine fit; ARC-raw skips removal.Source: §G.1 · p. 48

G.2 WHAT THE COMPARISON ESTABLISHES

  • Across the evaluated trajectory, residual-minus-raw differences range from −17.7 to +5.6 without a consistent advantage for residualization.At 100K, raw and residual variants improve on the unaligned model by 47.6 and 42.0 proxy points, about 21–24% of the matched REPA gain.Source: §G.2 · p. 49, Table 23 · p. 49
  • The control does not establish that pretrained encoders are necessary, because the self-teacher may provide too little useful structure.A stronger test would match encoder-free target quality to the pretrained target before comparing residualization.Source: §G.2 · p. 49

H.1 STANDARD EVALUATION AND MECHANISM CONTROLS

The controls show that improving prediction through alignment does not necessarily improve generation, while LAP’s gains are associated with encoder-target properties across the tested settings. The appendix also checks metric consistency, training trajectories, qualitative samples, and computational costs.

H.1 STANDARD EVALUATION AND MECHANISM CONTROLS

  • The auxiliary prediction loss improves velocity prediction but worsens generation, while the predictable-only target is worse than the unaligned baseline on standard FID, precision, and recall.The predictable-only target also has the lowest precision and recall among base-setting runs.Source: Table 24 · p. 49, §H.1 · p. 49
  • The appendix reports additional teacher, recipe, and tokenizer comparisons, including LAP-L results for DINOv3 and AIMv2 and LAP-N results with frozen alternative VAEs.AIMv2 LAP-L FID seeds are 44.40/44.22 versus raw-target 48.08/47.87; with MAE on the frozen REPA-E VAE, LAP-N seeds are 27.84/27.28 versus raw-target 33.42/34.06.Source: §H.1 · pp. 50–51

H.2 PRE-SPECIFIED ENCODER SWEEP AND AN EXTERNAL CHECK OF C0

  • Across 13 encoders, the LAP-L gain correlates with raw C0 at 0.94 at 100K steps and 0.79 at 400K; the association persists after controlling for raw-target FID.At 100K, the partial rank correlation is 0.83 with C0 given raw FID; at 400K it is 0.69. Gains below about 2 FID are within seed-to-seed noise.Source: §H.2 · pp. 52–53, Table 27 · p. 53, Figure 11 · p. 54

H.3 CHECKING THE DEVELOPMENT METRIC

  • The two development feature spaces have Spearman rank correlation 0.93, and both show a large improvement from purifying MAE, but close-method rankings can differ.This check concerns feature-space sensitivity, not independent training replication; individual DINOv2-proxy scores vary by about 1–15 points across sample draws.Source: §H.3 · p. 54, Table 28 · p. 55

H.4 COMPLETE TRAINING TRAJECTORIES

  • At 100K steps, LAP-L improves MAE, AIMv2, and CLIP by 10.7, 4.3, and 4.0 FID-10K, respectively.LAP-N with MAE scores 72.8 vs. 75.0 for DINOv2-based REPA at 100K and 45.1 vs. 45.1 at 400K; these are gains per training step, not measured total-compute savings.Source: Table 29 · p. 55, §H.4 · p. 55

H.5 QUALITATIVE SAMPLES

  • The qualitative comparisons use matched class labels and sampling noise within each column, while guidance-scale-4 samples illustrate rather than replace distribution-level metrics.The comparisons cover the five teachers, REPA-family recipes, frozen tokenizers, and SiT-XL/2 at batch 256.Source: §H.5 · p. 56, Figure 12 · p. 57, Figure 13 · p. 58

I COMPUTATIONAL RESOURCES

  • The experiments lack matched wall-clock and peak-memory benchmarks, so per-step convergence gains do not establish lower total compute.LAP-N evaluates its frozen predictor once per batch during training; the predictor is discarded for sampling.Source: I COMPUTATIONAL RESOURCES · p. 61

J TWO AERIAL-SCENE DATASETS

Development-scale tests on two aerial-scene datasets find that purified ImageNet MAE is the best aligned target in the reported comparisons, but the authors describe the evidence as suggestive. The related-work discussion distinguishes where alignment guidance enters the pipeline from how targets or matching objectives are transformed.

J TWO AERIAL-SCENE DATASETS

  • On RESISC45, purified ImageNet MAE has mean FID 61.5±2.3, while on AID it improves ImageNet MAE from 66.1±1.5 to 60.1±3.3.Purified ImageNet MAE is the best aligned target on both datasets, but margins over no alignment are at most about two standard deviations over two to four seeds, so the motivation remains suggestive.Source: Table 30 · p. 61, J TWO AERIAL-SCENE DATASETS · pp. 61–62

K.1 WHERE AND WHEN REPRESENTATIONS GUIDE GENERATION

  • Related methods vary where or when guidance is applied, whereas this study asks which content within a fixed target is useful for alignment.The discussion identifies REPA-E, representation autoencoders, SRA, LayerSync, U-REPA, HASTE, VA-REPA, and TARO as approaches that alter pipeline location or supervision timing and strength.Source: §K.1 · p. 62

K.2 TRANSFORMING THE TARGET OR MATCHING OBJECTIVE

  • The authors distinguish LAP’s focus on locally predictable content within a fixed encoder from iREPA’s focus on spatial structure, without claiming those explanations are mutually exclusive.Other cited work transforms targets or changes matching objectives, including PixelREPA, Spectrum Matching, SARA, and structural representation alignment.Source: §K.2 · p. 62

K.3 MECHANISTIC ANALYSIS AND DISTILLATION

The related-work discussion positions the paper’s mechanistic analysis as a study of target content and prediction interventions, rather than layer selection or a general theory of diffusion optimization. The appendix cautions that two runs do not provide a well-calibrated confidence interval.

K.3 MECHANISTIC ANALYSIS AND DISTILLATION

  • The paper distinguishes its analysis of target content and auxiliary prediction from AG-REPA’s layer interventions, and describes its linear-Gaussian analysis as illustrative rather than a theory of diffusion optimization.The analysis is intended to show why decodability and predictive contribution can differ.Source: §K.3 · pp. 62–63
  • LAP’s local predictors define operational residuals but do not guarantee removal of all reconstruction information; the cited self-teacher result applies only to its tested construction.The discussion also distinguishes LAP’s clean-latent prediction utility from knowledge-distillation objectives and specified linear concept-erasure criteria.Source: §K.3 · p. 63

L INDIVIDUAL RUNS FOR THE MAIN COMPARISONS

L.1 AUXILIARY PREDICTION SWEEP

The auxiliary prediction sweep shows that improved velocity prediction does not ensure better generation, and the appendix collects encoder-target and individual-seed results behind the main comparisons.

L.1 AUXILIARY PREDICTION SWEEP

  • At λs = 0.5, dev-200 velocity sufficiency is 0.50 versus 0.52 at λs = 2, while mean FID-DINOv2 worsens from 1159 to 1187 and then 1247.Reliance on the aligned subspace rises to 40% only at λs = 2; at full scale FID-50K worsens in both seeds, and degradation does not require greater reliance on supervised directions.Source: Table 31 · p. 63, §L.1 · p. 63, Figure 16 · p. 64

L.2 ENCODER TARGETS AND PURIFICATION

  • The appendix pairs clean-latent prediction utility for five raw encoders with development-scale generation results, and provides a development-scale target ladder.The DINOv2 linear split explains about 2.7% of centered variance.Source: Table 32 · p. 64, Table 33 · p. 64, §L.2 · p. 64

L.3 STANDARD FID BY SEED

  • The individual-seed tables provide standard FID-50K runs behind the main teacher, intervention, recipe, and tokenizer comparisons.The listed runs use the base SiT-B/2, batch-64, 400K recipe; larger-setting evaluations are reported separately.Source: Table 34 · p. 65, §L.3 · p. 65, Table 35 · p. 66

Related papers

Loading 2610.20761v1…