PaperlayerView exact revision on arXiv

Source-linked AI summary

What matters for Representation Alignment: Global Information or Spatial Structure?

Jaskirat Singh, Xingjian Leng, Zongze Wu, Liang Zheng, Richard Zhang, Eli Shechtman, Saining Xie

arXiv:2512.10794v1cs.CVcs.AIcs.GRcs.LGstat.ML

TL;DR

The paper investigates whether REPA’s generation benefits come from global semantic information or spatial structure in target representations. It analyzes diverse vision encoders, measures spatial structure, and introduces iREPA modifications that emphasize spatial transfer. Spatial structure better predicts generation performance, while iREPA consistently accelerates REPA convergence across encoders, model sizes, and training variants.

  • Problem

    It is unclear whether REPA benefits primarily from global semantic information or spatial structure, despite the common assumption that higher ImageNet-1K performance improves generation.

  • Method

    The paper analyzes 27 vision encoders and three model sizes using spatial-structure metrics, then introduces spatial regularization and convolutional projection to form iREPA.

  • Results

    Spatial structure metrics correlated with generation FID at Pearson |r| > 0.852, versus |r| = 0.26 for ImageNet-1K accuracy, and iREPA consistently improved REPA convergence speed.

  • Takeaways & Limitations

    Spatial structure, rather than global semantic performance, is the supported indicator of REPA generation effectiveness, motivating spatially focused alignment designs.

Abstract

from arXiv · show

Representation alignment (REPA) guides generative training by distilling representations from a strong, pretrained vision encoder to intermediate diffusion features. We investigate a fundamental question: what aspect of the target representation matters for generation, its \textit{global} \revision{semantic} information (e.g., measured by ImageNet-1K accuracy) or its spatial structure (i.e. pairwise cosine similarity between patch tokens)? Prevalent wisdom holds that stronger global semantic performance leads to better generation as a target representation. To study this, we first perform a large-scale empirical analysis across 27 different vision encoders and different model scales. The results are surprising; spatial structure, rather than global performance, drives the generation performance of a target representation. To further study this, we introduce two straightforward modifications, which specifically accentuate the transfer of \emph{spatial} information. We replace the standard MLP projection layer in REPA with a simple convolution layer and introduce a spatial normalization layer for the external representation. Surprisingly, our simple method (implemented in $<$4 lines of code), termed iREPA, consistently improves convergence speed of REPA, across a diverse set of vision encoders, model sizes, and training variants (such as REPA, REPA-E, Meanflow, JiT etc). %, etc. Our work motivates revisiting the fundamental working mechanism of representational alignment and how it can be leveraged for improved training of generative models. The code and project page are available at https://end2end-diffusion.github.io/irepa

1 INTRODUCTION

The paper asks whether REPA benefits primarily from global semantic information or spatial structure in target representations. Across diverse encoders, spatial structure better predicts generation, motivating iREPA, which accentuates spatial transfer and consistently improves convergence speed.

  • Motivation: REPA’s mechanism is unclear: improvements may reflect global semantic information or relationships between patch-token representations.The distinction matters for selecting target representations and maximizing representation-alignment benefits.
  • Evidence: Higher global accuracy did not guarantee better generation: PE-Spatial-B reached better REPA generation than PE-Core-G despite 53.1% versus 82.8% accuracy.The paper reports the same pattern for WebSSL-1B, which had 76.0% accuracy but worse generation than the 53.1% model.
  • Evidence: Across 27 vision encoders and three model sizes, spatial metrics correlated with generation FID far better than ImageNet-1K accuracy.Spatial metrics achieved Pearson |r| > 0.852, versus |r| = 0.26 for ImageNet-1K validation accuracy.
  • Outcome: iREPA consistently improves REPA convergence speed across vision encoders, model sizes, and training variants.Reported variants include REPA, REPA-E, Meanflow with REPA, and JiT with REPA.
  • Method: The authors introduce a spatial structure metric and modifications that accentuate spatial-information transfer from target representations to diffusion features.The modifications include spatial regularization and replacing the standard MLP projection with a convolution layer.

2 MOTIVATION: GLOBAL INFORMATION MATTERS LESS

Experiments challenge the assumption that stronger global semantic performance yields better REPA generation. Across encoder families and controlled token mixing, spatial structure is a better indicator of generation performance.

  • Cross-encoder comparisons: Higher ImageNet-1K accuracy does not imply better generation with REPA across target representations.PE-Core-G achieved 82.8% accuracy but FID 32.3, whereas PE-Spatial-B achieved 53.1% and FID 21.0.
  • Cross-encoder comparisons: SAM2-S achieved better REPA FID than encoders with substantially higher ImageNet-1K accuracy despite only 24.1% validation accuracy.The comparison includes PE-Core-G at 82.8% accuracy.
  • Model scaling: Larger encoders within the same family often produced similar or worse generation performance despite better validation accuracy.The paper reports this pattern for DINOv2, PE, and C-RADIO families.
  • Controlled global-information test: Increasing CLS-token mixing improved linear probing accuracy from 70.7% to 78.5% but worsened FID from 19.2 to 25.4.The mixing strength α increased from 0 to 0.5 while global information was added to local patch tokens.
  • Conclusion: The observations indicate that spatial structure provides a better signal for REPA performance than global accuracy.The paper presents this as the interpretation of the encoder comparisons and token-mixing experiments.

3 SPATIAL STRUCTURE MATTERS MORE

Spatial structure is a stronger predictor of generation performance than global semantic performance in representation alignment. Across encoders and model scales, spatial metrics consistently correlate more strongly with FID or gFID than linear probing.

  • Spatial structure metrics: Spatial self-similarity metrics measure how patch-token similarity varies with lattice distance, with larger values indicating stronger spatial organization.LDS contrasts similarity among nearby patches with similarity among distant patches.
  • Correlation analysis: Across 27 vision encoders, linear probing correlates weakly with FID, whereas all reported spatial metrics show much stronger correlations.Pearson |r| = 0.260 for linear probing versus LDS 0.852, SRSS 0.885, CDS 0.847, and RMSC 0.888.
  • Correlation analysis: Across SiT-B, SiT-L, and SiT-XL, spatial structure consistently correlates more strongly with generation performance than linear probing.Across model scales, linear probing has |r| < 0.306, while spatial structure has |r| > 0.826.
  • Explaining encoder behavior: PE-Spatial-B achieves better REPA generation despite lower ImageNet-1K accuracy than PE-Core-G, consistent with its stronger spatial structure.PE-Spatial-B versus PE-Core-G: accuracy 53.1% versus 82.8%, but FID 22.0 versus 32.3.
  • Explaining encoder behavior: Lower spatial structure accompanies worse generation when global information is increased through larger encoders or CLS-token mixing.The paper also reports that SIFT, HOG, and intermediate VGG features improve REPA, supporting benefits from spatial features alone.

4 iREPA: IMPROVING REPRESENTATION ALIGNMENT BY ACCENTUATING WHAT MATTERS

iREPA modifies REPA to transfer spatial information more faithfully from target representations to diffusion features. Its spatial normalization and convolutional projection consistently improve convergence or generation quality across encoders, scales, depths, and training variants.

  • Method: iREPA introduces spatial normalization and convolutional projection to accentuate spatial-feature transfer from teacher representations to diffusion features.The method is described as a straightforward modification implemented in fewer than four lines of code.
  • Method: Replacing REPA’s MLP with a 3×3 convolution preserves local spatial relationships on the patch grid and avoids observed loss of spatial contrast.The convolution uses kernel size 3 and padding 1.
  • Method: Spatial normalization improves patch-token contrast by reducing the global component of pretrained representations.The normalization computes expectation and variance across the spatial dimension, using epsilon = 10^-6 for numerical stability.
  • Generalization: iREPA consistently accelerates convergence across diverse vision encoders and model sizes, including SiT-XL/2 and SiT-B/2.The reported encoder set includes DINOv3-B, WebSSL-1B, PE-Core-G, CLIP-L, MoCov3, and PE-Lang-G.
  • Scalability: iREPA’s percentage improvement increases with encoder size, from 22.2% for PE-B to 38.8% for PE-L and 39.6% for PE-G.The paper also reports larger percentage gains with larger diffusion models and consistent improvements across alignment depths.
  • Ablations: Both spatial normalization and convolutional projection improve generation quality over REPA, with the best results obtained when combined.The ablation is reported at 100K steps with SiT-XL/2.
  • Training variants: The spatial improvements generalize across REPA-E, MeanFlow with REPA, classifier-free guidance settings, and pixel-space JiT diffusion.Faster convergence is reported both with and without classifier-free guidance and across multiple vision encoders for JiT.

5 RELATED WORK

Prior work studies representation alignment for accelerating diffusion training and examines the tradeoff between global and spatial information in pretrained vision encoders.

  • Representation alignment: Recent representation-alignment methods improve diffusion training by aligning internal diffusion features with clean image features from pretrained vision encoders.The section situates REPA among methods using external visual representations for generative modeling.
  • Global–spatial tradeoff: Related studies report that increased similarity between global CLS and patch tokens can worsen performance on dense spatial tasks.This work builds on broader research about the tradeoff between global and spatial information.

6 CONCLUSION

The paper concludes that spatial structure, rather than global information, drives representation-alignment effectiveness. It introduces iREPA as a simple spatial-transfer modification that consistently improves convergence across diverse settings.

  • Conclusion: The paper’s central conclusion is that spatial structure, not global information, drives representation-alignment effectiveness.This conclusion follows the paper’s large-scale empirical analysis.
  • Conclusion: iREPA applies spatial improvements across diverse encoders and training recipes, with reported results covering variation in training settings and classifier-free guidance.The supplied table descriptions summarize consistent gains over baseline REPA.
  • Implication: The paper motivates revisiting how representational alignment works and how spatial information can be leveraged in generative-model training.This is presented as a direction for future research rather than a claim beyond the reported experiments.

B SPATIAL SELF-SIMILARITY METRICS

The paper evaluates spatial self-similarity in patch-token representations using distance-based and semantic-region metrics. Larger metric values generally indicate stronger spatial organization or diversity, and iREPA improves generation across encoders and depths.

  • Spatial self-similarity framework: Spatial self-similarity measures how cosine similarity between patch tokens varies with lattice distance.Larger values indicate stronger spatial organization, with nearby patches more similar than distant patches.
  • Distance-based metrics: LDS contrasts average cosine similarity for nearby and distant patch pairs using rnear and rfar thresholds.Its default thresholds are rnear = rfar = H/2, and the reported correlation is robust to their exact choices.
  • Distance-based metrics: CDS fits a least-squares line to the spatial correlogram and uses the fitted slope to quantify similarity decay with distance.Larger CDS values indicate faster decay and stronger spatial organization.
  • Semantic-region metrics: SSM compares cosine similarity between anchor-positive pairs within a SAM2 semantic mask and anchor-negative pairs outside it.Larger SSM values indicate that patches in the same semantic region are more similar than patches from unrelated regions.
  • Contrast metrics: RMSC measures spatial contrast by computing the root-mean-square deviation of normalized patch features from their spatial mean.Higher RMSC indicates greater feature diversity and preserved spatial structure, whereas lower values indicate more uniform features.
  • Empirical behavior: iREPA consistently improves generation quality across vision encoders, SiT model sizes, and encoder depths.The depth comparison reports slightly better performance with 8 layers than with 6 layers, while both configurations show significant FID improvements.

E.1 SIT-B/2 RESULTS

The SiT-B/2 and SiT-L/2 evaluations compare iREPA with vanilla REPA across vision encoders at 400K iterations. The tables report consistent generation-quality improvements from iREPA.

  • SiT-B/2: iREPA consistently improves generation quality across vision encoders for SiT-B/2 at 400K iterations.The baselines use vanilla REPA for training.
  • SiT-L/2: iREPA consistently improves generation quality across vision encoders for SiT-L/2 at 400K iterations.The gains are particularly strong for FID and IS metrics, with vanilla REPA used for all baselines.

F ADDITIONAL EXPERIMENTAL RESULTS

Additional experiments examine global information in patch tokens, classifier-free guidance, convergence speed, and spatial-information transfer. Together, these results emphasize preserving spatial structure during alignment.

  • Global information: Mean patch tokens contain substantial global semantic information, but removing some of it can improve spatial structure transfer.Global information increases global performance while reducing contrast between individual patch tokens.
  • Classifier-free guidance: iREPA consistently improves both IS and FID with classifier-free guidance across vision encoders.The comparison uses SiT-XL/2, CFG scale 2.0, 400K iterations, and 250 sampling steps.
  • Convergence speed: Accentuating spatial features improves convergence speed for SiT-XL/2 with iREPA compared with REPA.The figure directly compares convergence results for the two alignment methods.
  • Spatial transfer: Replacing the standard MLP projection with a convolution better preserves spatial information during transfer to diffusion features.The figure also motivates spatial normalization because pretrained vision features often have limited spatial contrast.

G IMPLEMENTATION DETAILS

The experiments follow REPA’s ImageNet-based training and ADM-based preprocessing and evaluation protocols. Implementations adapt existing REPA variants, while the paper also describes related generative-modeling and vision work.

  • Training setup: Training uses ImageNet images center-cropped and resized to 256 × 256, with the stabilityai/sd-vae-ft-mse VAE throughout diffusion training and inference.Spatial normalization uses γ ∈ [0.6, 0.8].
  • Implementations: REPA-E and JiT use official open-source implementations, while MeanFlow adapts an author-provided implementation for iREPA.The adaptations introduce iREPA’s two changes.
  • Evaluation: Generation quality is evaluated with gFID, sFID, IS, precision, and recall on 50K generated images.Sampling follows REPA with an SDE Euler-Maruyama sampler and 250 steps.
  • Related work: Pretrained visual encoders have been used as discriminators, adversarial-distillation teachers, and alignment targets in generative modeling.The related work frames representation alignment as a recent use of pretrained encoder features.
  • Related work: Transformer generative models include diffusion, flow-matching, and autoregressive denoising formulations, but training from scratch remains computationally expensive.REPA is described as accelerating convergence, while this paper attributes its key benefit to preserving spatial structure.
  • Denoising and representation learning: Prior denoising-based representation-learning methods motivate the view that denoising naturally encourages robust features.The paper positions its findings as complementary, emphasizing strong spatial representations during external alignment.
  • Vision encoders: The paper studies scaled self-supervised vision encoders, including DINOv3 and WebSSL, trained on very large unlabeled image collections.These works represent the broader scaling trend in self-supervised visual encoding.

J ADDITIONAL DISCUSSION ON SPATIAL STRUCTURE METRICS

Spatial structure metrics help explain representation alignment and correlate more strongly with generation performance than global semantic measures. Spatial normalization and spatial-feature transfer consistently improve convergence across encoders, resolutions, tasks, and model sizes.

  • Spatial structure metrics: Spatial structure metrics can measure the representation gap between pretrained encoder and diffusion features.Representation alignment helps close the spatial-structure gap between DINOv2 and diffusion features.
  • Spatial structure metrics: Spatial structure correlates more strongly with generation quality than linear probing accuracy across 27 vision encoders.For SiT-XL-2 and SiT-B-2, LDS–FID correlations are −0.85 and −0.89, versus −0.26 and −0.12 for linear probing accuracy.
  • Motivating examples: Higher global accuracy can accompany worse generation: PE-g has 82.8% accuracy and 32.3 FID, versus Spatial-PE-g’s 72.4% and 22.0.PE-g also has lower spatial structure, with LDS 0.1 versus 0.4.
  • Spatial normalization: Spatial normalization removes global overlays and enhances local token contrast, producing clearer separation between semantic regions.The transformation reduces correlations between unrelated regions while preserving local spatial structure.
  • Generalization: Accentuating spatial-feature transfer improves convergence speed across ImageNet resolutions, multimodal text-to-image tasks, and pretrained encoder choices.Reported settings include ImageNet-256 and ImageNet-512, with encoders such as DINOv2, CLIP, WebSSL, and PE.
  • Robustness: After removing outliers, spatial structure remains more correlated with generation performance than linear probing across model sizes B, L, and XL.The reported correlations are Pearson |r| > 0.85 for spatial structure and Pearson |r| < 0.38 for linear probing.

M.5 ADDITIONAL RESULTS WITH ALTERNATIVE EVALUATION METRICS

The paper tests whether its conclusions hold under alternative generation metrics and broader evaluation settings. CMMD reproduces the stronger spatial-structure relationship, while iREPA improves convergence across metrics, with marginal gains for SAM2.

  • Alternative metrics: CMMD provides an alternative evaluation metric for testing the robustness of the representation-alignment findings.The analysis complements Inception Score, FID, sFID, precision, and recall.
  • Alternative metrics: Pearson |r| > 0.88 for spatial metrics versus Pearson |r| = 0.074 for linear probing accuracy when predicting CMMD generation performance.The comparison uses SiT-B/2 trained for 100K steps with REPA.
  • iREPA results: Across vision encoders, iREPA improves convergence speed under CMMD and traditional metrics, both with and without classifier-free guidance.Traditional metrics include IS, FID, sFID, precision, and recall.
  • SAM2: For SAM2-S, spatial normalization yields only marginal improvements because the encoder already contains little global information.SAM2-S has validation accuracy below 24%, leaving less global component for normalization to remove.
  • SAM2: The study uses intermediate SAM2 encoder outputs rather than mask logits as target representations.Mask logits may produce sharper spatial maps but are not suitable target representations in this setting.
  • Accuracy measure: Full-finetuning accuracy produces similar overall findings, but linear probing remains the default global-information measure for REPA.The paper notes that finetuning can alter the global information in encoder features used for regularization.
  • Pixel-space diffusion: Combined with JiT, iREPA outperforms recently proposed pixel-space diffusion methods in the reported 200K-iteration evaluation.The comparison uses 50-step Euler sampling without classifier-free guidance.
Loading 2512.10794v1…