Source-linked AI summary
DecQ: Detail-Condensing Queries for Enhanced Reconstruction and Generation in Representation Autoencoders
Tianhang Wang, Yitong Chen, Wei Song, Zuxuan Wu, Min Li, Jiaqi Wang
TL;DR
Frozen VFMs provide semantically rich latent representations for RAEs but under-represent low-level details, creating a reconstruction–generation trade-off. DecQ adds detail-condensing queries from intermediate VFM features, incorporates them into decoding, and jointly denoises them with patch tokens. Experiments report improved reconstruction and generation with minimal computational overhead and faster convergence than the frozen-VFM baseline.
Problem
Frozen VFMs preserve high-level semantics but under-represent low-level visual details needed for faithful reconstruction, while reconstruction-oriented augmentation can interfere with semantic representations.
Method
DecQ uses a small set of learnable queries with cross-attention to intermediate layers of a frozen VFM, then incorporates the queries into decoding and jointly denoises them with patch tokens.
Results
DecQ improves both reconstruction and generation, with faster convergence and better generation quality than the frozen-VFM baseline under the reported experimental settings.
Takeaways & Limitations
Condensing complementary information from shallow and deep VFM layers can improve reconstruction and generation simultaneously without changing the pretrained VFM latent space.
Takeaways & Limitations
Experiments are mainly conducted on ImageNet at 256 × 256 resolution, leaving generality on more diverse datasets insufficiently evaluated.
Abstract
from arXiv · showhide
Representation Autoencoders (RAEs) leverage frozen vision foundation models (VFMs) as tokenizer encoders, providing robust high-level representations that facilitate fast convergence and high-quality generation in latent diffusion models. However, freezing the VFM inherently constrains its spatial reconstruction capacity, limiting fine-grained generation and image editing; in contrast, incorporating reconstruction-oriented signals via fine-tuning disrupts the pretrained semantic space and degrades generative fidelity. To address this trade-off, we propose DecQ, a simple yet effective framework for RAEs. Specifically, DecQ introduces lightweight detail-condensing queries that extract fine-grained information from intermediate VFM features through condenser modules. These queries are incorporated into the decoder to support reconstruction and are jointly generated with patch tokens during generative modeling. By aggregating information from both shallow and deep layers, DecQ effectively mitigates the reconstruction--generation trade-off, improving both reconstruction quality and generative performance. Our experiments demonstrate that: (1) with only 8 additional queries and 3.9% extra computation, DecQ improves reconstruction over the frozen DINOv2-based RAE, increasing PSNR from 19.13 dB to 22.76 dB; and (2) for generative modeling, DecQ achieves 3.3$\times$ faster convergence than RAE, attaining an FID of 1.41 without guidance and 1.05 with guidance.
1 Introduction
RAEs use frozen VFM representations that preserve semantics but omit low-level visual detail, creating a reconstruction–generation trade-off. DecQ addresses this by adding detail-condensing queries that enrich reconstruction while preserving the frozen semantic space and supporting generation.
- Motivation: Frozen VFM representations under-represent low-level cues, causing texture loss, color shifts, and weaker fine-grained generation and editing.VFM objectives emphasize semantic robustness rather than pixel-level reconstruction.
- Motivation: Reconstruction-oriented alternatives improve fidelity but consistently degrade generative performance under the unified comparison.The compared variants include VFM fine-tuning, distillation, and feature concatenation.
- DecQ: DecQ uses learnable queries that attend to intermediate features of a frozen VFM to capture complementary low-level details.The queries enrich fine-grained information without modifying VFM parameters or perturbing its semantic representations.
- DecQ: DecQ jointly denoises detail-condensing queries with image patches during generation, allowing the added details to support generative modeling.The framework is designed to resolve the reconstruction–generation dilemma rather than optimizing reconstruction alone.
- Contributions: Condensing information from both shallow and deep VFM layers improves reconstruction and generation simultaneously with limited additional overhead.Shallow layers mainly benefit reconstruction, while deep layers benefit generation.
2 Related work
Related work improves diffusion generation through representation alignment, semantically aligned visual tokenizers, and direct use of VFMs as latent encoders. These approaches motivate DecQ’s focus on preserving VFM semantics while recovering missing visual detail.
- Representation Alignment in Diffusion Models: Representation-alignment methods accelerate diffusion training by aligning noisy diffusion states with clean VFM representations.Subsequent works refine this alignment framework in complementary directions.
- VFM-Aligned Visual Tokenizers for Generation: VFM-aligned visual tokenizers aim to build latent spaces with strong semantics by aligning or augmenting visual tokenizer representations.Examples include VA-VAE, AlignTok, and DMVAE.
- VFMs as Direct Tokenizers for Generation: RAE directly adopts VFMs as latent encoders, using their high-dimensional semantic space to achieve faster convergence and improved generation performance.Related approaches also modify VFM-based latent spaces to improve reconstruction.
3 Method
DecQ augments frozen VFM patch-token latents with compact detail-condensing queries extracted from intermediate features. These queries support reconstruction in the decoder and are jointly modeled with patch tokens for generation while preserving the VFM semantic space.
- Motivation: The method addresses RAE’s limitation that VFM patch tokens encode semantics well but under-represent low-level details needed for faithful reconstruction.This limitation includes color fidelity and fine-grained textures.
- Encoder with Condensers: DecQ adds learnable queries to intermediate layers of a frozen VFM, using condenser modules to aggregate complementary fine-grained information.The queries are compact because their number K is much smaller than the number of patch tokens N.
- Encoder with Condensers: Cross-attention lets queries read intermediate patch features while preventing those patches from altering the pretrained VFM representations.Patch tokens serve as keys and values, so information flows unidirectionally from patches to queries.
- Dual-Stream Decoder: The decoder concatenates semantic patch tokens and detail-condensing query tokens, using query self-attention to provide fine-grained details while only patch tokens produce pixels.Patch and query tokens use separate projections and positional embeddings before joint decoder processing.
- Generation with Detail-Condensing Queries: During generation, DecQ forms one latent sequence from patch and query tokens, jointly denoises it, and decodes both token types into the output image.The sequence is modeled with the DiTDH architecture under a flow-matching objective.
4 Experiments
Experiments show that DecQ improves reconstruction and generation while mitigating their trade-off, with faster convergence and effectiveness across query counts, VFM layers, and backbone choices.
- Experimental Setup: 8 detail-condensing queries with condensers at layers 0, 3, 6, and 9 provide the default configuration.The sparse placement balances reconstruction fidelity, generation quality, and computational cost.
- Reconstruction: DecQ achieves the lowest rFID among VFM-based tokenizers and substantially improves pixel-level reconstruction over RAE.The reported reconstruction evaluation uses PSNR, SSIM, and rFID at 256 × 256 resolution.
- Generation: DecQ attains FID 1.41 at 800 epochs without guidance and 1.05 with guidance, outperforming previous state-of-the-art methods.It also reaches FID 1.80 at 80 epochs.
- Convergence: DecQ reaches gFID 1.80 after 80 epochs and 1.51 after 240 epochs, matching RAE trained for 800 epochs and yielding 3.3× faster convergence.The comparison uses FID-50K on ImageNet at 256 × 256 resolution.
- Ablation Study: Predicting detail-condensing queries improves patch-token generation even when query tokens are discarded at inference, while retaining them further improves decoding quality.This comparison separates the auxiliary benefit of query prediction from the direct decoding benefit of generated queries.
- Ablation Study: Increasing query count consistently improves reconstruction, but 8 queries provide the best balance because excessive queries can interfere with generation.More queries may introduce redundant low-level information despite improving reconstruction.
- Ablation Study: Shallow VFM layers favor reconstruction, deeper layers favor generation, and DecQ improves both by preserving semantic structure while adding fine-grained information.The same framework remains effective with SigLIP2, indicating generality across VFMs.
- Analysis: Query and patch tokens encode complementary information: queries capture low-level appearance details, whereas patches preserve high-level semantics.The clustering study associates query tokens with color and texture and patch tokens with semantic content.
5 Conclusion
DecQ recovers fine-grained information lost in VFM representations while preserving their semantic latent space. It jointly denoises query and patch tokens to improve reconstruction and generation with minimal computational overhead.
- Conclusion: DecQ uses cross-attention to condense fine-grained information from intermediate VFM layers into detail-condensing queries.The queries recover information progressively lost in VFM representations.
- Conclusion: Jointly denoising query and patch tokens enables richer details and higher generation quality while preserving the VFM semantic latent space.Experiments consistently show improvements in both reconstruction and generation with minimal computational overhead.
- Conclusion: The decoder processes query and patch tokens jointly, while only patch tokens are used for final image reconstruction.Query tokens participate in decoder self-attention as auxiliary latent variables that enhance decoding.
A.2 Diffusion Model Implementation
The diffusion implementation follows RAE with LightningDiT, continuous-time flow matching, and standard sampling and guidance procedures. Training uses fixed optimization settings and evaluates 50,000 class-uniform samples.
- Diffusion Backbone: LightningDiT serves as the diffusion backbone with continuous-time flow matching over timesteps in [0, 1].Gaussian Fourier feature embeddings replace standard timestep embeddings.
- Diffusion Backbone: DiTDH-XL is used for main results and DiTDH-S for ablations, with a linear projection when backbone and DDT-head dimensions differ.The projection maps encoder outputs to the decoder dimension.
- Optimization: Optimization uses AdamW, a constant learning rate of 2.0 × 10^-4, batch size 1024, EMA decay 0.9999, and gradient clipping at 1.0.All diffusion models are trained on 8 NVIDIA H200 GPUs.
- Sampling and Evaluation: Sampling uses an Euler ODE solver with 50 default steps, while 250 steps can further improve results.FID-50K evaluation samples 50 images per class for 50,000 total images.
- Guidance: AutoGuidance uses DiTDH-S as the weaker guiding model, with the best results obtained from its 60-epoch checkpoint at guidance scale 1.6.This follows the minimal guidance variant used by RAE.
- Additional Evaluation: Additional experiments include clustering analyses, reconstruction comparisons on DINOv2 and SigLIP2, and qualitative generated samples.These results are presented as supplementary qualitative analyses.
B.1 Cluster Analysis
Cluster analyses distinguish the roles of DecQ query and patch tokens: queries preserve fine-grained appearance, while patches preserve high-level semantics. Qualitative reconstructions show stronger preservation of colors, text, and textures.
- Cluster Analysis: Query-token clusters share color and texture patterns, indicating that queries primarily capture fine-grained visual details.The analysis associates queries with low-level appearance information.
- Cluster Analysis: Patch-token clusters share semantic properties such as object identity and main subjects, indicating that patches preserve high-level semantics.Patch-token neighbors are grouped by semantic content rather than low-level appearance.
- Reconstruction Comparison: Compared with DINOv2-based RAE, DecQ better preserves background colors, textual content, and fine-grained textures in reconstruction.The qualitative comparison uses the same setting as Fig. 1 Right.
- Reconstruction Comparison: With SigLIP2, DecQ better preserves fine-grained colors when the corresponding RAE retains textual semantics but fails to reproduce them faithfully.This provides an additional reconstruction comparison across VFMs.
- Generation: Qualitative class-to-image samples demonstrate DecQ's strong generative capabilities.The samples are presented in Fig. 10.
C Computational Overhead Analysis
The analysis examines DecQ’s additional computational cost and presents qualitative reconstruction and generation results. DecQ’s reconstruction comparison highlights improved preservation of fine-grained details over SigLIP2-based RAE.
- DecQ’s overhead is analyzed alongside qualitative image reconstruction and generation results.The overhead analysis follows the MACs convention.
- DecQ better preserves fine-grained details than SigLIP2-based RAE in some reconstruction cases.SigLIP2 may retain textual semantic impressions but fail to reproduce colors accurately.
- Qualitative results of DecQ’s image generation are presented.
C.1 Tokenizer and Reconstruction Overhead
DecQ adds modest tokenizer and reconstruction overhead through condensers and additional query tokens. Its default configuration adds 3.9% computation and 5.8% parameters over the baseline.
- 3.9% additional computation is incurred by DecQ’s default tokenizer configuration.Four condensers extract eight query tokens, while the longer decoder sequence contributes additional computation.
- 5.8% additional parameters are introduced by DecQ relative to the 501.9M-parameter baseline.The added trainable parameters total 29.3M, including 28.36M from four condensers.
- The overhead table reports tokenizer and reconstruction costs using M for condenser-equipped VFM layers and K for query tokens.GFLOPs use the MACs convention, and active parameters count parameters participating in forward computation.
C.2 Generation Overhead
DecQ introduces modest generation overhead when query tokens are jointly processed during diffusion sampling and final decoding. The default setting adds 3.22% computation and 3.37M parameters.
- 3.22% additional computation is required for DecQ’s default generation setting.The baseline totals 8,189.8 GFLOPs, while query-token processing adds 263.4 GFLOPs across sampling and decoding.
- 3.37M additional parameters are introduced during generation with K=8 query tokens.Generation uses lightweight embedding and projection layers rather than tokenizer-stage cross-attention condensers.
- Additional generation computation scales approximately linearly with the number of query tokens.The quadratic attention term remains negligible in the evaluated regime.
D Limitations
DecQ’s evaluation is concentrated on ImageNet at 256 × 256 resolution and primarily uses DINOv2 and SigLIP2. Its generality and scaling at broader datasets, higher resolutions, and other backbones remain insufficiently studied.
- DecQ is mainly evaluated on ImageNet at 256 × 256 resolution.More diverse datasets, including text-to-image and domain-specific collections, are needed to assess generality more comprehensively.
- Higher-resolution generation at 512 × 512 or above has not been extensively studied.The scaling of query count, condenser placement, and computational overhead remains to be examined.
- Evaluation primarily covers DINOv2 and SigLIP2 as representative VFMs.Broader studies across backbone families and model scales remain an important direction, while denser configurations may increase cost.