Source-linked AI summary
Both Semantics and Reconstruction Matter: Making Representation Encoders Ready for Text-to-Image Generation and Editing
Shilong Zhang, He Zhang, Zhifei Zhang, Chongjian Ge, Shuchen Xue, Shaoteng Liu, Mengwei Ren, Soo Ye Kim, Yuqian Zhou, Qing Liu, Daniil Pakhomov, Kai Zhang, Zhe Lin, Ping Luo
TL;DR
Representation encoders are semantically rich but lack compact regularization and sufficient pixel fidelity for reliable generation. The paper introduces PS-VAE, which jointly uses semantic and pixel reconstruction in a compact latent space, achieving state-of-the-art reconstruction and strong text-to-image and editing performance. The authors also identify an architectural limitation in which a symmetric encoder-decoder design improves reconstruction but degrades GenEval.
Problem
Representation-encoder features are not directly suitable as generative spaces because unconstrained distributions produce off-manifold latents and weak reconstruction limits accurate geometry and texture.
Method
PS-VAE maps representation features and pixel details into a compact, KL-regularized latent space by fine-tuning encoders with semantic and pixel reconstruction objectives.
Results
PS-VAE achieves state-of-the-art reconstruction and generation, with rFID improving from 0.534 to 0.203 and instruction-based editing reward from 0.06 to 0.22 over MAR-VAE.
Takeaways & Limitations
Representation encoders can be adapted into unified components for visual understanding, text-to-image generation, and image editing.
Takeaways & Limitations
A symmetric architecture that feeds reconstructed semantic features directly into the pixel decoder improves reconstruction but degrades GenEval, possibly because shared objectives interfere.
Abstract
from arXiv · showhide
Modern Latent Diffusion Models (LDMs) typically operate in low-level Variational Autoencoder (VAE) latent spaces that are primarily optimized for pixel-level reconstruction. To unify vision generation and understanding, a burgeoning trend is to adopt high-dimensional features from representation encoders as generative latents. However, we empirically identify two fundamental obstacles in this paradigm: (1) the discriminative feature space lacks compact regularization, making diffusion models prone to off-manifold latents that lead to inaccurate object structures; and (2) the encoder's inherently weak pixel-level reconstruction hinders the generator from learning accurate fine-grained geometry and texture. In this paper, we propose a systematic framework to adapt understanding-oriented encoder features for generative tasks. We introduce a semantic-pixel reconstruction objective to regularize the latent space, enabling the compression of both semantic information and fine-grained details into a highly compact representation (96 channels with 16x16 spatial downsampling). This design ensures that the latent space remains semantically rich and achieves state-of-the-art image reconstruction, while remaining compact enough for accurate generation. Leveraging this representation, we design a unified Text-to-Image (T2I) and image editing model. Benchmarking against various feature spaces, we demonstrate that our approach achieves state-of-the-art reconstruction, faster convergence, and substantial performance gains in both T2I and editing tasks, validating that representation encoders can be effectively adapted into robust generative components.
1 Introduction
Representation encoders offer semantically rich features but are not directly effective generative spaces because unconstrained latents cause off-manifold errors and weak reconstruction loses fine details. The paper addresses both issues with compact semantic-pixel latents and reports strong reconstruction, generation, and editing performance.
- 1 Introduction: Representation-space generation suffers from off-manifold latents caused by insufficient compact regularization, producing inaccurate and structurally distorted objects.The mismatch between high feature dimensionality and lower intrinsic information content makes diffusion training prone to unreliable latent regions.
- 1 Introduction: Weak pixel-level reconstruction in discriminative encoders prevents generators from learning accurate geometry, small-scale structure, and texture.The paper addresses this by jointly optimizing pixel and semantic reconstruction objectives after unfreezing the encoder.
- 1 Introduction: PS-VAE achieves state-of-the-art reconstruction and generation, while improving rFID from 0.534 to 0.203 and editing reward from 0.06 to 0.22 over MAR-VAE.It also improves PSNR from 26.18 to 28.79, SSIM from 0.715 to 0.817, GenEval from 75.8 to 76.6, and DPG-Bench from 83.2 to 83.6.
- 1 Introduction: PS-VAE converts representation features into a compact 96-channel latent space using semantic reconstruction and adds pixel reconstruction to preserve structural and textural details.The approach is instantiated with DINOv2 and is also validated on SigLIP2.
- 1 Introduction: The study establishes a controlled evaluation pipeline across generation-space designs and evaluates text-to-image and instruction-based editing performance.The framework is used to demonstrate PS-VAE’s superior performance under unified evaluation.
2 Related work
Visual understanding encoders provide discriminative semantic features, whereas VAEs provide compact pixel-oriented latents for visual generation. This work connects these spaces by designing a compact, KL-regularized latent representation informed by high-level semantics.
- 2 Related work: Representation encoders support visual understanding tasks through discriminative features, while VAEs reduce generation cost using compact latents focused on pixel-level structure.The paper positions these spaces as complementary foundations for understanding and generation.
- 2 Related work: The proposed latent space combines VAE-style generative stability with the discriminative power of foundation-model features through explicit semantic conditioning and KL regularization.This design extends efforts to bridge discriminative and generative feature spaces.
- 2 Related work: RAE provides semantically rich features and stronger prompt following for image editing, but its weak reconstruction causes detail loss and severe artifacts in text-to-image generation.Its generation performance substantially lags behind VAE despite its semantic advantages.
3 Method
The analysis identifies off-manifold generation and weak pixel reconstruction as complementary obstacles when using representation features for generation. The proposed PS-VAE regularizes these features into a compact semantic-pixel latent space, improving structure, texture, reconstruction, and downstream generation.
- Analysis of RAE: Ambient 8D diffusion produces more and farther off-manifold samples than diffusion in the intrinsic 2D space.The toy experiment embeds a 2D PS-shaped distribution into 8D and measures nearest-neighbor distances for the top 5% tail samples.
- Analysis of RAE: PS-VAE addresses off-manifold generation by converting unconstrained representation features into a compact latent space before diffusion.The analysis links redundant high-dimensional features to off-manifold latents and structurally distorted objects, motivating compact regularization.
- Analysis of RAE: Equation 1 decomposes the high-dimensional denoiser into intrinsic-manifold flow plus an orthogonal residual, forcing ambient models to discover the manifold and transmit nonsemantic noise.The manifold component requires implicit projection and embedding, while the orthogonal component imposes an identity-like mapping burden.
- Make Representation Encoders Ready: PS-VAE uses a 96-channel semantic VAE with semantic reconstruction and KL regularization, then adds pixel reconstruction to preserve fine-grained details.The semantic stage freezes the representation encoder and trains on semantic losses; after convergence, the encoder is unfrozen so pixel reconstruction gradients enrich structure and texture.
- Make Representation Encoders Ready: The progressively improved latent spaces RAE → S-VAE → PS-VAE reduce artifacts and improve texture and structure, while preserving semantics for generation and editing.The framework reports improved reconstruction quality alongside faster text-to-image coverage and stronger instruction-following ability for editing.
- Generation Architecture: The selected TransFusion-style block improves parameter efficiency relative to Bagel-style fusion, whose 1.1-point gain requires 71% more parameters.LlamaFusion exhibits a bottleneck attributed to its frozen language branch, whereas the chosen design is used as the core fusion architecture.
4 Experiments
Across reconstruction, generation, editing, and encoder-transfer evaluations, PS-VAE provides a semantically structured, high-fidelity latent space with strong benchmark performance and faster convergence.
- 4.2 Reconstruction and Generation Performance of Different Feature Space: PS-VAE96c achieves the highest reconstruction quality among stride-16 VAEs and outperforms RAE variants across generation and editing evaluations.PS-VAE32c leads DPG-Bench and Editing Reward, while PS-VAE96c leads GenEval; both converge faster than RAE and other VAEs.
- 4.3 Scaling Behavior of Generative Models across PS-VAE Channel Dimensions: GenEval rises from 76.56 to 78.14 and Editing Reward from 0.222 to 0.285 when PS-VAE96c scales from Qwen-0.5B to Qwen-1.5B.DPG-Bench also increases from 83.62 to 84.09, whereas PS-VAE32c shows marginal or negative changes on these tasks.
- 4.4 PS-VAE with SigLIP2: A Unified Encoder for Understanding and Generation: PS-VAE96c with SigLIP2 achieves comparable reconstruction and generation capabilities to the DINOv2 version, supporting transfer across pretrained foundation encoders.SigLIP2 performs slightly better on GenEval, while DINOv2 performs better on DPG-Bench and Editing Reward.
- 4.4 PS-VAE with SigLIP2: A Unified Encoder for Understanding and Generation: Replacing Bagel’s original encoder with the fine-tuned encoder causes negligible zero-shot understanding degradation while enabling the unified reconstruction and generation framework.The reported MME-P score decreases slightly from 1685 to 1652.
5 Ablation Study
The ablations show that semantic regularization addresses off-manifold generation, pixel supervision restores detail fidelity, and approximately 96 channels balance both requirements under the tested setup.
- 5.1 Evolution from RAE to PS-VAE: GenEval improves from 71.3 to 73.7 and Editing Reward from 0.06 to 0.12 when RAE becomes S-VAE, despite PSNR falling from 19.2 to 17.78.The ablation attributes the generation gain to compact semantic regularization mitigating off-manifold features.
- 5.1 Evolution from RAE to PS-VAE: PS-VAE combines semantic regularization with pixel-detail supervision, achieving state-of-the-art reconstruction while improving generation and editing over the RAE-to-S-VAE progression.Its editing reward nearly doubles from 0.12 to 0.22 while preserving semantic coherence and improving detail fidelity.
- 5.1 Evolution from RAE to PS-VAE: 83.6 to 82.6 DPG-Bench and 0.22 to 0.04 Editing Reward after removing semantic structure show that pixel-only reconstruction is insufficient for semantic alignment.Linear probing also returns P-VAE’s semantic quality to the MAR-VAE level.
- 5.2 Ablation on Latent Channel of PS-VAE: Approximately 96 channels provide the best joint trade-off: performance remains comparable from 32 to 96 channels, while DPG-Bench drops by about 0.6 points at 112 channels.Further capacity mainly adds high-frequency details, slows convergence, and may hinder semantic alignment.
- 5.3 Ablation on Encoder and Decoder Architectures: A shallow 2-layer MLP fails to preserve rich semantic features in the 96-channel latent, causing posterior collapse under KL regularization and degraded generation.The results indicate that mapping capacity must capture the representation’s intrinsic dimensionality.
- 5.4 High-Dimensional Enrichment: Direct high-dimensional pixel enrichment improves reconstruction but produces severe structural artifacts and incoherent textures because shortcut reconstruction bypasses meaningful latent geometry.The corresponding high-dimensional enrichment variant is summarized as improving reconstruction while harming generation.