Source-linked AI summary
The Prism Hypothesis: Harmonizing Semantic and Pixel Representations via Unified Autoencoding
Weichen Fan, Haiwen Diao, Quan Wang, Dahua Lin, Ziwei Liu
TL;DR
Multimodal systems struggle to share semantic meaning while preserving pixel-level detail because semantic and pixel representations emphasize different frequency content. The paper proposes the Prism Hypothesis and UAE, which organize a shared latent space into semantic and residual detail bands; experiments report unified performance across latent and pixel spaces.
Problem
Existing semantic and pixel encoders follow separate trajectories, and their heterogeneous representations create a tension between abstraction and fidelity during unification.
Method
UAE uses frequency decomposition and a hierarchical frequency-band modulator to place semantic structure in low-frequency bands and pixel detail in residual higher-frequency bands.
Results
UAE outperforms RAE, SVG, and UniFlow across rFID, PSNR, gFID, and Accuracy on ImageNet while preserving reconstruction quality comparable to Flux-VAE.
Takeaways & Limitations
The Prism Hypothesis provides a shared spectral perspective for unified understanding and generation, and UAE operates in both latent and pixel spaces.
Takeaways & Limitations
The paper formalizes semantic-alignment behavior under the assumption that alignment is carried by a shared low-frequency base.
Abstract
from arXiv · showhide
Deep representations across modalities are inherently intertwined. In this paper, we systematically analyze the spectral characteristics of various semantic and pixel encoders. Interestingly, our study uncovers a highly inspiring and rarely explored correspondence between an encoder's feature spectrum and its functional role: semantic encoders primarily capture low-frequency components that encode abstract meaning, whereas pixel encoders additionally retain high-frequency information that conveys fine-grained detail. This heuristic finding offers a unifying perspective that ties encoder behavior to its underlying spectral structure. We define it as the Prism Hypothesis, where each data modality can be viewed as a projection of the natural world onto a shared feature spectrum, just like the prism. Building on this insight, we propose Unified Autoencoding (UAE), a model that harmonizes semantic structure and pixel details via an innovative frequency-band modulator, enabling their seamless coexistence. Extensive experiments demonstrate that UAE effectively unifies semantic abstraction and pixel-level fidelity within a single latent space, achieving state-of-the-art performance. Moreover, we show that UAE can be directly applied to pixel-space modeling, significantly improving both FID and IS over the vanilla JIT baseline. Our code is avaliable at: https://github.com/WeichenFan/UAE.
1. Introduction
Existing systems separate semantic abstraction from pixel fidelity, creating representational conflict and leaving unified modeling unresolved. The Prism Hypothesis and UAE address this tension by linking encoder spectra to a shared latent space that preserves both meanings and details.
- 1. Introduction: Semantic and pixel encoders excel at different objectives, forcing unified systems to reconcile heterogeneous representations.Semantic encoders capture high-level meaning, whereas pixel encoders compress fine-grained visual detail.
- 1. Introduction: The mismatch lowers training efficiency and causes incompatible features to interfere rather than complement one another.
- 1. Introduction: Prior efforts improve semantic correspondence or add semantic awareness to pixel encoders, but commonly achieve coexistence through trade-offs rather than integration.Semantic transfer can limit fine-detail recovery, while text supervision, distillation, and hierarchical integration do not fully eliminate the trade-off.
- 1. Introduction: The Prism Hypothesis proposes that semantic features occupy low-frequency bands while pixel features extend into higher-frequency bands representing fine appearance and geometry.These complementary representations can be integrated within a unified encoder across a continuous shared feature spectrum.
- 1. Introduction: UAE factorizes content into a fundamental semantic band and residual pixel bands, producing a shared latent space that outperforms RAE, SVG, and UniFlow across reconstruction and perception metrics.The reported metrics are rFID, PSNR, gFID, and Accuracy on ImageNet.
2. Related Work
Related work develops unified representations, multi-resolution generation, and spectral analysis, while UAE explicitly organizes continuous latent features into semantic and detail bands. Frequency-filtering evidence supports the view that semantic alignment is concentrated in low frequencies.
- Unified Tokenizers and Unified Representations: UAE explicitly factorizes continuous latent features into a low-frequency base for semantics and residual high-frequency bands for progressive detail reconstruction.
- Frequency and Multi-Resolution Modeling: Frequency-energy analysis shows DINOv2 and CLIP emphasize low-frequency semantic content, whereas SD-VAE retains more high-frequency energy for finer details.
- Frequency and Multi-Resolution Modeling: Prior frequency-aware methods use pyramids, wavelets, Fourier features, periodic activations, frequency-aware losses, and alias-free synthesis to model structure across scales.
- Frequency and Multi-Resolution Modeling: Cascaded diffusion, feature pyramids, hierarchical patch schedules, and next-granularity or next-frequency generation progressively model global layout and fine detail.
- Frequency and Multi-Resolution Modeling: Text–image retrieval remains stable after low-pass filtering but degrades sharply after high-pass filtering, indicating semantic alignment primarily resides in low-frequency components.
3. Methodology
UAE unifies semantic structure and pixel detail by organizing representations into frequency bands and jointly optimizing semantic alignment with high-fidelity reconstruction. The method supports both frequency-domain and spatial-domain generative modeling.
- Prism Hypothesis: The Prism Hypothesis treats semantic encoders as emphasizing shared low-frequency structure, while pixel encoders retain additional high-frequency modality-specific detail.The shared low band is associated with categories, attributes, and relations; higher bands encode edges, textures, and fine appearance.
- Unified Autoencoding: UAE jointly optimizes low-frequency semantic alignment and pixel-wise reconstruction, harmonizing semantic structure with visual detail in one latent space.The architecture uses a pretrained semantic encoder, a trainable unified encoder, and a pixel decoder with spectral transform blocks.
- Empirical Verification: Low-pass retrieval supports the hypothesis: Recall@1 rises from approximately 0.23 at 0.05 Nyquist to approximately 0.58 at 0.10 as low-frequency content is retained.The experiment evaluates text–image retrieval after progressively removing high-frequency image components.
- Generative Modeling: UAE latents support two generative parameterizations: direct frequency-domain modeling and spatial-domain modeling after inverse DCT.The frequency-wise formulation progressively models frequency tokens from low to high bands, while the spatial-wise formulation diffuses after transforming them back to spatial features.
- Frequency Representation: UAE decomposes latents with channel-wise 2D DCT, then reorders coefficients into a frequency-progressive token sequence using zig-zag traversal.The ordering approximately increases frequency magnitude with token position, supporting explicit low-to-high frequency processing.
- Semantic Regularization: Semantic regularization aligns only the first Kbase low-frequency bands, leaving higher-frequency bands less constrained for modality-specific pixel detail.Randomly masking high-frequency tokens trains the decoder to reconstruct missing fine details from preserved low-frequency structure and visible coefficients.
4. Experiments
UAE is evaluated for reconstruction and class-conditional generation, with results indicating strong fidelity, perceptual quality, and compatibility with efficient latent modeling. Experiments also examine qualitative detail preservation and convergence-related comparisons.
- Visual Reconstruction: 31.00 PSNR, 0.92 SSIM, and 0.35 FID on ImageNet-1K improve over RAE’s 18.05, 0.50, and 2.04, respectively.On MS-COCO, UAE reaches 31.19 PSNR, 0.91 SSIM, and 2.01 FID versus RAE’s 18.36, 0.47, and 6.01.
- Visual Reconstruction: UAE preserves straight edges, fine textures, and small text across natural photos and illustrations, while working with DINOv2, CLIP, and SigLIP2 encoders.
- Generative Modeling: 1.52 gFID and 234.5 IS on ImageNet-1K place UAE on par with existing state-of-the-art generative models.The experiments use a multi-band latent space and report class-conditional generation at 256 × 256 resolution.
- Generative Modeling: UAE converges faster and reaches lower FID than SiT and RAE, with the final gap highlighted at 80 epochs.
5. Discussion
The discussion attributes UAE’s behavior to frequency-aware representations: low-frequency tokens carry semantic information, while unified training and masking improve detail modeling and generation efficiency. These properties extend from latent-space modeling to pixel-space JIT.
- 5.1. The Impact of Generative Modeling: UAE’s frequency-aware representation accelerates diffusion convergence and yields the best FID at 80 epochs compared with SiT and RAE.Both spatial and frequency variants achieve lower FID throughout training and perform strongly as early as 40 epochs.
- 5.2. Efficiency: 313.84 to 78.80 GFLOPs and 8.82 to 4.05 ms per image show the efficiency gain from using 64 low-frequency UAE tokens.
- 5.3. The Impact of Semantic Loss: Aligning 25% of frequency tokens preserves most semantic capability while maintaining high reconstruction quality; excessive semantic constraints reduce reconstruction fidelity.
- 5.4. Frequency-Masked Prediction: Masking 85% of high-frequency tokens achieves the lowest gFID@20 epochs of 9.7, supporting aggressive masking for generative modeling.
- 5.5. Frequency Analysis: Unified training shifts energy toward mid- and high-frequency bands while only slightly reducing low-frequency dominance associated with global semantics.
- 5.5. Frequency Analysis: Generative performance favors preserved low-frequency tokens, while training from scratch shows semantic information emerging during compression and reconstruction.
- 5.6. UAE in Pixel Space: Pixel-space and latent-space modeling are presented as differing mainly in compression ratio and compression architecture, enabling UAE’s extension to pixel space.
- 5.6. UAE in Pixel Space: UAE consistently improves both FID and IS over vanilla JIT across pixel-space training stages under identical configurations.
6. Conclusion
The paper proposes the Prism Hypothesis as a shared spectral view of natural inputs and introduces UAE to unify semantic and pixel information. Experiments support UAE as a practical unified tokenizer across latent and pixel spaces.
- The Prism Hypothesis represents natural inputs as a compact low-frequency semantic component plus residual higher-frequency detail.
- UAE uses a hierarchical frequency-band modulator to harmonize semantic and pixel information in one latent space, improving generation while preserving reconstruction quality.The paper positions UAE as effective in both latent-space and pixel-space settings.