PaperlayerView exact revision on arXiv

Source-linked AI summary

REGLUE Your Latents with Global and Local Semantics for Entangled Diffusion

Giorgos Petsangourakis, Christos Sgouropoulos, Bill Psomas, Theodoros Giannakopoulos, Giorgos Sfikas, Ioannis Kakogeorgiou

arXiv:2512.16636v2cs.CV

TL;DR

Latent diffusion provides only indirect semantic supervision, while existing VFM-guided approaches use a narrow slice of available semantics. REGLUE jointly models VAE latents with compressed local and global VFM semantics in one diffusion model, and on ImageNet 256×256 reaches ReDi’s and REG’s 1M-step performance using less than 30% and 80% of their iterations, respectively.

  • Problem

    Existing LDM and VFM approaches provide only indirect denoising semantics or expose narrow, non-spatial or linear VFM features, under-utilizing multi-layer spatial semantics.

    Source: §1 · pp. 1–2

  • Method

    REGLUE jointly models VAE latents, compressed local patch-level VFM semantics, and a global [CLS] token within a single SiT model.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    On ImageNet 256×256, SiT-XL/2+REGLUE reaches the 1M-step performance of ReDi and REG using less than 30% and 80% of their iterations, respectively.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Experiments indicate that spatial patch-level semantics are the primary source of gains, while a global token and external alignment provide complementary enhancements.

    Source: §1 · p. 2

    Evaluation is limited to 1M iterations for SiT-XL/2 and omits 512×512 ImageNet generation, which the authors identify as a next step.

    Source: C Limitations and Future Work · pp. 29–30

Abstract

from arXiv · show

Latent diffusion models (LDMs) achieve state-of-the-art image synthesis, yet their reconstruction-style denoising objective provides only indirect semantic supervision: high-level semantics emerge slowly, requiring longer training and limiting sample quality. Recent works inject semantics from Vision Foundation Models (VFMs) either externally via representation alignment or internally by jointly modeling only a narrow slice of VFM features inside the diffusion process, under-utilizing the rich, nonlinear, multi-layer spatial semantics available. We introduce REGLUE (Representation Entanglement with Global-Local Unified Encoding), a unified latent diffusion framework that jointly models (i) VAE image latents, (ii) compact local (patch-level) VFM semantics, and (iii) a global (image-level) [CLS] token within a single SiT backbone. A lightweight convolutional semantic compressor nonlinearly aggregates multi-layer VFM features into a low-dimensional, spatially structured representation, which is entangled with the VAE latents in the diffusion process. An external alignment loss further regularizes internal representations toward frozen VFM targets. On ImageNet 256x256, REGLUE consistently improves FID and accelerates convergence over SiT-B/2 and SiT-XL/2 baselines, as well as over REPA, ReDi, and REG. Extensive experiments show that (a) spatial VFM semantics are crucial, (b) non-linear compression is key to unlocking their full benefit, and (c) global tokens and external alignment act as complementary, lightweight enhancements within our global-local-latent joint modeling framework. The code is available at https://github.com/giorgospets/reglue .

1 Introduction

REGLUE jointly models compact spatial VFM semantics, a global [CLS] token, and VAE latents in one diffusion framework. Its introduction argues that learned nonlinear compression preserves richer multi-layer spatial semantics than earlier narrow semantic signals.

1 Introduction

  • LDM denoising supervises semantics only indirectly while learning high-level meaning and low-level visual detail together, so semantic structure emerges slowly and limits quality and convergence.Pretrained VFM representations have been used to accelerate convergence and improve image quality.Source: §1 · pp. 1–2
  • Earlier approaches use either a non-spatial global [CLS] token or linearly projected patch features, while REGLUE targets richer nonlinear, multi-layer spatial semantics.The introduction contrasts REG’s image-level signal with ReDi’s linear PCA-projected local features; it notes that spatial semantics can also be distilled through REPA’s external alignment loss.Source: §1 · p. 2
  • The introduction identifies joint modeling of compact patch features with VAE latents as the primary source of gains, with global [CLS] and external alignment as complementary signals.It states that spatial guidance yields larger gains than external feature alignment alone or joint modeling of a single image-level token.Source: §1 · p. 2
  • REGLUE jointly models patch-level and global VFM semantics with VAE latents, using a lightweight compressor to map multi-layer VFM features into a compact, semantics-preserving space.A feature-alignment auxiliary loss aligns diffusion internal features with VFM teacher representations.Source: §1 · p. 2
  • On ImageNet 256×256, SiT-XL/2+REGLUE reaches ReDi’s and REG’s 1M-step performance using less than 30% and 80% of their iterations, respectively.The paper also reports that local semantics, global semantics, and REPA-style alignment improve image quality and convergence while leaving model parameters and inference-time compute essentially unchanged.Source: §1 · p. 3

2 Related Work

Related work connects latent-variable generative models and semantic representation learning, while prior approaches either align diffusion representations to VFMs or jointly model VFM features. REGLUE instead entangles global and local VFM semantics with VAE latents in one diffusion backbone.

Latent-variable generative modeling

  • VAEs and diffusion share variational foundations, and prior work has analyzed their links to flows and probabilistic PCA; REGLUE applies unified modeling to VAE latents and VFM semantics.The passage also situates SiT, DiT, and Lightning DiT within modern generative frameworks.Source: §2 · p. 3

Representation alignment with VFM features

  • Prior work aligns VAE latents or denoisers to VFMs, whereas REGLUE jointly models global and local VFM semantics with VAE latents in one diffusion backbone.Examples include VA-VAE, REPA, and REPA-E; REGLUE uses a frozen, lightweight semantic compressor rather than exposing only a narrow semantic slice or relying solely on external alignment.Source: §2 · p. 3

Joint feature generative modeling

  • REG models compressed tokens and a VFM [CLS] token, while ReDi jointly diffuses spatial high-level features and VAE latents; REGLUE identifies limitations in both approaches.The paper argues that REG discards useful spatial cues, whereas ReDi uses constrained linear projections of high-level semantics; related examples include CoDi and VideoJam [9].Source: §2 · pp. 3–4

Representation learning

  • Visual representation learning has progressed from supervised CNNs and handcrafted self-supervised tasks to contrastive, self-distillation, masked-image, and vision-language pretraining.CLIP aligns images and captions contrastively; SigLIP uses independent sigmoid losses, and SigLIP-2 [70] further improves transfer through data and training strategies.Source: §2 · p. 4

3 Method

REGLUE jointly models VAE latents, compressed local VFM semantics, and a global [CLS] token in one SiT backbone. A frozen, nonlinear spatial compressor makes patch-level semantics compact, while an auxiliary alignment loss matches internal representations to VFM targets.

3.1 Preliminaries

  • SiT uses a stochastic interpolant between clean VAE latents and Gaussian noise, trains a Transformer velocity field, and by default uses a linear schedule.The default schedule is α_t = 1 − t and σ_t = t.Source: §3.1 · p. 4

3.2 Global–local representation entanglement

  • REGLUE jointly diffuses VAE latents, compressed patch-level VFM features, and a global [CLS] token using a shared noise schedule and multimodal velocity objective.The three modalities receive independent noise terms, and the model predicts a velocity for each.Source: §3.2 · pp. 5–6
  • The model merges projected latent and local-semantic features channel-wise into one patch grid, while retaining the global [CLS] representation as a separate token.This avoids the 2×-longer sequence and quadratic self-attention overhead of concatenating separate latent and semantic patch tokens.Source: §3.2 · p. 6
  • At a selected SiT block, a lightweight projector applies cosine alignment between hidden tokens and frozen global and patch-level VFM targets.Alignment covers the [CLS] position and semantic patch tokens and is added to the multimodal velocity loss.Source: §3.2 · p. 7

3.3 A lightweight spatial semantic compressor

  • A shallow convolutional autoencoder compresses multi-layer VFM patch features into frozen, spatially structured semantics, addressing the dimensional imbalance of naïve fusion.The compressed features are resampled to the VAE latent grid before joint modeling.Source: §3.3 · pp. 7–8

4 Experiments

On ImageNet 256×256, REGLUE improves diffusion generation quality and convergence by jointly modeling compressed local VFM features with global signals, while experiments identify nonlinear spatial compression as central to preserving useful semantics. Ablations show complementary benefits from global tokens and external alignment, and compressor choices affect both semantic retention and diffusability.

4.1 Setup

  • Experiments use ImageNet 256×256 and SD-VAE-FT-EMA latents; SiT-B/2 trains for 400K steps and SiT-XL/2 for 1M, with batch size 256.Evaluation uses 50,000 generated samples and reports FID, sFID, IS, Precision, and Recall.Source: §4.1 · p. 8

4.2 Analyzing VFM semantics for generation

  • With nonlinear compression, local patch modeling reaches 14.3 FID without alignment, and aggregating the last four VFM blocks before compression reaches 12.9 FID in final REGLUE.The multi-layer local-only setting reaches 13.3 FID; adding alignment and global [CLS] modeling improves the progression to 14.1 and 13.7 FID, respectively. DINOv3-B yields 12.3 FID, compared with 12.9 for default DINOv2-B.Source: Table 1 · p. 9, §4.2 · pp. 10–11

4.3 Enhancing diffusion models

  • At 400K steps, REGLUE reduces SiT-B/2 FID by 60.9% versus vanilla SiT-B/2, 47.1% versus REPA, and 39.7% versus ReDi.At 300K steps, REGLUE reaches 14.5 FID, surpassing REG’s 15.2 FID at 400K; throughput and inference-time parameters remain unchanged.Source: Table 2 · p. 11, §4.3 · p. 11
  • REGLUE achieves 28.7 FID in unconditional generation, improving over SiT-B/2 (59.8), ReDi (43.6), and REG (29.7).The passage reports improvements of 52%, 34.2%, and 3.4% over SiT-B/2, ReDi, and REG, respectively.Source: Table 2 · p. 11, §4.3 · p. 11
  • At 160 epochs, REGLUE reaches 1.46 FID versus REG’s 1.59, while 80-epoch REGLUE reaches 1.59 and matches 160-epoch REG.Despite using 5× fewer epochs than 800-epoch variants, 160-epoch REGLUE remains competitive with REPA (1.42 FID) and REG (1.36 FID).Source: §4.3 · pp. 11–12, Table 4 · p. 12
  • At 1M steps, REGLUE achieves 2.5 FID, besting REG (2.7), ReDi (5.1), and REPA (6.4); at 700K, it reaches 2.7 FID, matching REG’s 1M-step performance.Source: §4.3 · p. 11

4.4 Semantic compressor impact

  • On ImageNet, with 8 channels the nonlinear semantic compressor achieves substantially higher attentive probing accuracy and lower FID than PCA-compressed ReDi features; 16 channels further improves both, approaching full 768-channel DINOv2.Four channels degrade FID; performance improves up to 16 channels and degrades again at 20, so REGLUE uses 16 channels.Source: §4.4 · p. 13
  • Compressing DINOv2 layers 9–12 reaches 13.3 FID, versus 14.3 for layer 12 alone and 16.9 when shallow layers 3 and 6 are included.The compressor also suppresses PCA’s high-frequency tail toward an RGB-like frequency profile while preserving semantic signal, yielding a more diffusion-friendly representation.Source: §4.4 · pp. 14–15, Figure 3 · p. 15

5 Conclusion · A Additional Experimental Results

REGLUE jointly models VAE latents with compressed VFM patch-level and global semantics, improving generation FID and accelerating convergence on ImageNet baselines.

5 Conclusion

  • REGLUE entangles reconstruction-optimized VAE latents with patch-level and global VFM semantics using lightweight compression and aggregation, improving ImageNet generation FID and convergence.The conclusion characterizes these gains as significant margins but supplies no quantitative values.Source: §5 · p. 15

A.1 Semantic preservation under compression · A.2 Semantic compressor regularization · A.3 Impact of VFM

Nonlinear compression retains VFM semantics while improving generative fidelity over PCA, and plain MSE outperforms KL regularization. Among tested VFMs, DINOv3-B gives the lowest FID, while DINOv2-B remains the adopted default.

A.1 Semantic preservation under compression

  • At 8 channels, nonlinear compression scores 67.1 mIoU/14.3 FID versus PCA’s 59.1 mIoU/21.4 FID; at 16 channels it reaches 68.7 mIoU/13.3 FID.These compressed features use 96× or 48× fewer channels than the original 768-channel DINOv2 representation while retaining most semantics and improving generative fidelity.Source: §A.1 · pp. 21–22, Figure 5 · p. 23
  • The default compressor concatenates the VFM’s last four layers, projects them to 16 channels, and trains a semantic de-compressor with MSE reconstruction loss.Its layers are Conv2D(3072, 256), ResidualBlock(256, 256), and Conv2D(256, 16).Source: Figure 4 · p. 22

A.2 Semantic compressor regularization

  • Increasing the compressor’s KL weight consistently worsens gFID and ImageNet linear-probing accuracy, while precision and recall stay flat; plain MSE performs best.The ablation adds a variational KL term to an MSE-only autoencoder and models local semantics without external alignment during diffusion training.Source: §A.2 · pp. 22–23, Table 8 · p. 22

A.3 Impact of VFM

  • The VFM comparison uses the last four encoder layers, 16-channel compressed features, and 25 compressor-training epochs; each downstream SiT-B/2 generator trains for 400K steps.The input projection is adapted to each encoder’s embedding size for a fair comparison.Source: §A.3 · pp. 23–24
  • REGLUE’s qualitative samples reach high fidelity early in training, with the figure showing SiT-B/2+REGLUE at 50K, 100K, 200K, and 400K steps.The snapshots use identical noise and sampling schedules and step counts, without classifier-free guidance.Source: Figure 6 · p. 24
  • DINOv3-B delivers the lowest FID, DINOv2-B is a close second, and CLIP-L lags; REGLUE adopts DINOv2-B for consistency with prior work.Source: Table 9 · p. 23, §A.3 · p. 24

A.4 Detailed benchmark

REGLUE improves FID with training, reaching 2.5 FID at 1M steps after already surpassing the vanilla SiT-XL/2 baseline at 100K steps.

A.4 Detailed benchmark

  • 7.8 FID at 100K steps surpasses vanilla SiT-XL/2’s 8.3 FID at 7M; REGLUE then reaches 3.2 FID at 400K, 2.6 FID at 750K, and 2.5 FID at 1M.Table 10 reports FID, sFID, inception score, precision, and recall; the supplied passage gives values only for FID.Source: §A.4 · p. 24
  • Figure 6 shows high-fidelity REGLUE generations early in training.Source: §A.4 · p. 24

A.5 Additional Metrics · A.6 Classifier-free guidance · A.7 Small-scale & high-resolution dataset.

REGLUE’s gains extend beyond Inception-based metrics and to Food-101 and ImageNet-512 without classifier-free guidance. The appendix also evaluates guidance scales and intervals for SiT-XL/2+REGLUE at 800K steps.

A.5 Additional Metrics

  • At Base scale, REGLUE outperforms ReDi and REG across CMMD, FD, and FD∞, indicating its gains are not an artifact of Inception features.Table 11 reports the alternative metrics on ImageNet 256×256 without CFG.Source: §A.5 · p. 24, Table 11 · p. 25
  • At XL scale, REGLUE at 1M iterations matches or surpasses REG at 2.4M iterations on the reported metrics, using only 42% of its training iterations.The comparison is reported for CMMD, FD, and FD∞ on ImageNet 256×256 without CFG.Source: §A.5 · p. 24, Table 11 · p. 25

A.6 Classifier-free guidance

  • For SiT-XL/2+REGLUE at 800K steps, the appendix evaluates CFG scales and guidance intervals on ImageNet 256×256.The reported results are presented in Table 12.Source: §A.6 · p. 25
  • The CFG scale w is applied to both VAE latents and VFM representations, with a VAE-Only setting applying CFG exclusively to VAE latents.The guidance interval is varied over [0, τ], following.Source: §A.6 · p. 25

A.7 Small-scale & high-resolution dataset.

  • On Food-101 and ImageNet-512, SiT-B/2+REGLUE consistently outperforms established baselines in FID without classifier-free guidance.The evaluation tests robustness on a small-scale, different-domain dataset and a high-resolution dataset.Source: §A.7 · p. 25
  • The robustness evaluation uses Food-101 as a different-domain, small-scale dataset and ImageNet-512 as a high-resolution dataset, both with SiT-B/2.The comparison is made without CFG and reports FID scores.Source: §A.7 · p. 25

A.8 FID vs Training Compute · A.9 Fusion strategy. · A.10 Resampling VFM grid

REGLUE improves FID over REG at matched training compute without sequence-length overhead, while channel-wise fusion offers a better efficiency-to-quality trade-off than sequence concatenation. Bilinear resampling performs best among the tested interpolation kernels, with negligible difference from the nearest alternative.

A.8 FID vs Training Compute

  • REGLUE’s gains do not come from extra token or compute budget: it processes the same number of tokens as REG, with negligible overhead from the semantic compressor.The compressor adds DINO-v2 and semantic-compressor computation, but the reported total overhead is negligible.Source: §A.8 · pp. 25–26
  • Channel-wise fusion combines local semantics with VAE latents along channels rather than concatenating them along the sequence, avoiding sequence-length overhead.This is the design used to keep REGLUE’s token count equal to REG’s.Source: §A.8 · p. 25

A.9 Fusion strategy.

  • The sequence-level concatenation ablation uses 513 tokens: 256 VAE latents, 256 patch-level semantic features, and one global [CLS] token.This contrasts with the proposed channel-wise fusion strategy.Source: §A.9 · p. 26
  • Concatenation yields a negligible 4% FID gain but cuts throughput 55% (3.8 to 1.7 images/s), so channel-wise fusion remains the default.The throughput loss is attributed to quadratic attention cost; channel-wise fusion is reported to have the better efficiency-to-quality trade-off.Source: §A.9 · p. 26

A.8 FID vs Training Compute

  • At XL scale, REGLUE matches REG’s 1M-step FID at 700K steps—30% less compute.REGLUE also consistently outperforms REG at every matched compute point.Source: §A.8 · p. 26, Table 13 · p. 27

A.10 Resampling VFM grid

  • Bilinear resampling achieves the best performance when resizing VFM features to the latent grid, with negligible difference from the nearest alternative.The comparison evaluates interpolation kernels for resizing Eψ(f∗) to the Hz×Wz grid.Source: §A.10 · p. 27

A.11 Limited data

REGLUE improves on REG in limited-data ImageNet training, with larger FID gains at lower data fractions.

A.11 Limited data

  • -5.5 FID at 20% and -3.4 at 50%: REGLUE outperforms REG more strongly as ImageNet data becomes scarce.SiT-B/2 was trained for 80 epochs on class-balanced ImageNet sets of 20%, 50%, and 100%.Source: §A.11 · p. 27

B Additional Experimental Setup · B.1 Semantic compressor details · B.2 SiT details

The appendix specifies the semantic compressor’s architecture and training, then gives the SiT model configurations and optimization setup.

B.1 Semantic compressor details

  • The semantic encoder maps concatenated VFM features from 3072 channels to 256, applies a 256-channel residual block, and projects to 16 compressed channels.The input and output layers are 3×3 convolutions.Source: §B.1 · pp. 27–28
  • The fully convolutional compressor preserves spatial resolution, uses a symmetric 16→256→3072 de-compressor, and retains only its encoder at inference.It is trained to reconstruct VFM features with an MSE loss.Source: §B.1 · p. 28
  • 25 epochs of compressor training use MSE and Adam with learning rate 1 × 10^-3, batch size 4096, and no weight decay.The learning rate follows a cosine schedule to 8.5 × 10^-4; (β1, β2) = (0.9, 0.999).Source: §B.1 · p. 28
  • Under one hour on 8×A100 GPUs, compressor training loss decreases smoothly and plateaus by the final epoch.Source: §B.1 · p. 28

B.2 SiT details

  • SiT-B/2 has 132M parameters, 12 transformer blocks, embedding dimension 768, and 12 attention heads; SiT-XL/2 has 677M parameters, 28 blocks, dimension 1152, and 16 heads.The configurations follow the official SiT settings.Source: §B.2 · p. 28
  • Both SiT models use AdamW at constant learning rate 1 × 10^-4 and batch size 256, with fp16 mixed precision and gradient clipping.Image latents are precomputed using SD-VAE; training uses v-prediction and an Euler–Maruyama sampler with 250 steps, α_t = 1 − t, and σ_t = t.Source: §B.2 · p. 28

C Limitations and Future Work · D Baseline Generative Models · E Visualizations

The appendices report REGLUE’s ImageNet subset results, practical scaling limits and future directions, summarize comparison baselines, and show uncurated generations.

C Limitations and Future Work

  • On ImageNet subsets of 20%, 50%, and 100%, REGLUE improves over REG by −5.5, −3.4, and −2.3 FID, respectively.The paper also reports that REGLUE improves sample quality under comparable training budgets and reaches or surpasses strong baselines in fewer iterations.Source: C Limitations and Future Work · p. 28, Figure 7 · p. 29
  • Resource limits restrict SiT-XL/2+REGLUE experiments to 1M iterations, leaving full convergence at schedules such as 4M unexplored.On 8×A100 GPUs, one 1M-iteration run takes roughly 7 days, making longer schedules impractical.Source: C Limitations and Future Work · p. 29
  • ImageNet 512×512 experiments, such as with SiT-XL/2, remain a proposed next step; this work instead prioritizes practical settings including limited-data regimes.The limited-data results are shown in Figure 7.Source: C Limitations and Future Work · p. 30
  • Proposed extensions include stronger VFMs, a compact global-token compressor, and decoding semantic and VAE latents jointly.The authors report that DINOv3 yields further gains, while preliminary joint-decoding trials had mixed results across model scales.Source: C Limitations and Future Work · p. 30

D Baseline Generative Models

  • Comparisons cover autoregressive, latent-diffusion, and visual-representation methods, including VAR [68], SiT, REPA, ReDi, and REG.Other listed baselines are MagViTv2 and MAR; LDM, DiT, U-ViT-H/2, MaskDiT [87], MDT [21], SD-DiT, and FasterDiT. REPA aligns diffusion features with local VFM features; ReDi linearly compresses and models VFM features; REG models global VFM representations with REPA-style alignment.Source: D Baseline Generative Models · pp. 30–31

E Visualizations

Loading 2512.16636v2…