PaperlayerView exact revision on arXiv

Source-linked AI summary

UniSpace: Unified Visual Representation and Scalable Multimodal Modeling

Jinbo Yan, Limeng Qiao, Jie Qin, Junyan He, Feize Wu, Guanglu Wan

arXiv:2608.08676v1cs.CVcs.AI

TL;DR

Semantic vision encoders lose fine-grained details needed for reconstruction-sensitive tasks, motivating a shared representation for understanding, generation, and editing. The paper introduces Patch Reparameterization, which adds a reconstruction-aware patch pathway to a frozen semantic ViT, and scales it into UniSpace. UniSpace uses this unified visual space for multimodal tasks, achieving practical generation and editing while preserving semantic capabilities.

  • Problem

    Semantic encoder tokens provide visual semantics but have poor pixel-level recoverability, limiting their use as the sole interface for detail-sensitive generation and editing.

  • Method

    Patch Reparameterization preserves the original semantic pathway and adds a reconstruction-aware patch embedding to the same frozen ViT, then scales it into the UniSpace multimodal model.

  • Results

    UniSpace supports multimodal understanding, high-quality image editing, and practical text-to-image generation in a single visual space at 8B model scale.

  • Takeaways & Limitations

    A reparameterized pretrained semantic ViT can serve as a unified visual interface without a separate VAE pathway.

  • Takeaways & Limitations

    UniSpace’s system-level understanding performance lags behind dedicated vision–language models, and updating the shared encoder on understanding data may compromise generation.

Abstract

from arXiv · show

Semantic vision encoders have become a central visual interface for multimodal understanding and semantic conditioning in image generation. However, their final tokens discard fine-grained visual details, leading to poor pixel reconstruction and limiting their use in reconstruction-sensitive tasks such as image generation and editing. In this work, we ask whether understanding, generation, and editing can be modeled in a single visual representation space built from a pretrained semantic ViT. We show that the frozen Transformer blocks of a semantic ViT are not intrinsically unable to preserve visual details. Instead, the original patch parameterization drives the representation toward semantic abstraction, making fine-grained information difficult to recover from the final tokens. Based on this observation, we introduce \emph{Patch Reparameterization}, which preserves the original semantic pathway while adding a reconstruction-aware patch embedding that provides fine-grained visual information to the same frozen ViT blocks. The resulting unified representation preserves multimodal understanding while enabling high-fidelity image reconstruction and a favorable reconstruction--generation trade-off. We further scale this representation into \emph{UniSpace}, an 8B Mixture-of-Transformer-Experts model that performs understanding, generation, and editing in the same visual space without a separate VAE pathway. System-level evaluations demonstrate practical text-to-image generation and instruction-based image editing, showing that a reparameterized pretrained ViT can serve as a unified visual interface for scalable multimodal modeling.

1 INTRODUCTION

Semantic encoders provide strong visual semantics but discard fine-grained details needed for reconstruction-sensitive generation and editing. UniSpace addresses this by reparameterizing a pretrained semantic ViT and scaling it into a unified multimodal system.

  • Motivation: Semantic encoder tokens are optimized for abstraction, causing poor pixel-level recoverability in detail-sensitive generation and editing.This motivates a visual interface that preserves both semantic guidance and fine-grained image information.
  • Research question: The paper asks whether understanding, generation, and editing can share one representation built on a pretrained semantic ViT.The design must retain the pretrained semantic backbone while adding reconstruction capability and supporting scale.
  • Method: Patch Reparameterization keeps the original semantic pathway and adds a reconstruction-aware patch embedding to the same frozen Transformer blocks.The two token streams are merged into a unified representation trained for understanding and reconstruction.
  • Scaling: UniSpace scales PR-Qwen-ViT into an 8B Qwen-based Mixture-of-Transformer-Experts model using the patch-reparameterized encoder–decoder as its sole visual interface.This removes the need for a separate visual pathway within the model.
  • Results: UniSpace supports strong text-to-image generation and instruction-based editing while retaining general visual understanding capabilities under unified generative training.The system uses one visual space for understanding, generation, and editing.

2 RELATED WORK

Prior work extends semantic representations toward reconstruction and explores unified multimodal models, but visual representations remain heterogeneous. UniSpace uses a patch-reparameterized pretrained ViT as the sole visual tokenizer across tasks.

  • Semantic representations: Semantic representations improve generative modeling, but they are not automatically unified tokenizers for understanding, reconstruction, and generation.Recent methods use distillation or multi-layer features to extend semantic representations toward reconstruction.
  • Unified multimodal models: Unified multimodal models support understanding, generation, and editing through varied discrete or continuous visual representation designs.A common design uses separate pretrained visual spaces for different pathways.
  • UniSpace: UniSpace preserves a pretrained ViT semantic prior and uses its patch-reparameterized encoder–decoder as the only visual tokenizer.Understanding, generation, and editing operate in one representation space without a separate VAE latent space.

3 UNIFIED ENCODER VIA PATCH REPARAMETERIZATION

Patch Reparameterization exposes reconstruction-relevant information through an additional input pathway while preserving the original semantic ViT pathway. Explicit fusion then creates one shared representation for understanding, reconstruction, and generation.

  • 3.1 PATCH REPARAMETERIZATION CAN CARRY VISUAL DETAILS: Semantic ViT final tokens are poorly recoverable because the original patch parameterization suppresses fine-grained details, not because frozen Transformer blocks cannot carry them.A random input projection improves deep-layer recoverability while leaving the Transformer blocks unchanged.
  • 3.1 PATCH REPARAMETERIZATION CAN CARRY VISUAL DETAILS: Random patch embedding raises last-layer PSNR from 20.96 to 24.66 with the same frozen Transformer blocks.At the patch-embedding output, pretrained and random projections achieve PSNR values of 39.29 and 39.68, respectively.
  • 3.2 CONSTRUCTING A UNIFIED REPRESENTATION: Patch Reparameterization retains the original semantic embedding and adds a reconstruction-aware embedding that uses the same frozen Transformer blocks.The additional path supplies visual details without replacing the pretrained semantic representation.
  • 3.2 CONSTRUCTING A UNIFIED REPRESENTATION: Figure 2 combines semantic and reconstruction-aware tokens through Token Fusion into a unified representation shared by understanding, reconstruction, and generation.The semantic pathway is preserved while the second pathway injects fine-grained information into the frozen backbone.
  • 3.2 CONSTRUCTING A UNIFIED REPRESENTATION: The reconstruction tokens are linearly compressed before concatenation with semantic tokens, reducing generative-model burden while retaining decoder-critical details.Explicit channel separation keeps semantic and reconstruction information separately addressable.
  • 3.2 CONSTRUCTING A UNIFIED REPRESENTATION: The unified representation is used for all downstream tasks, while reconstruction training updates only the added patch embedding, projection, and decoder.The original semantic embedding and pretrained ViT blocks remain frozen, structurally preserving semantic tokens.
  • 3.3 TRAINING FOR RECONSTRUCTION AND GENERATION: Balanced flow matching normalizes semantic and reconstruction components by dimension and assigns an explicit objective weight during generative modeling.The channel-wise decomposition enables direct control over the relative training emphasis of both components.

4 UNISPACE: SCALING UNIFIED MODELING IN ONE VISUAL SPACE

UniSpace scales unified visual modeling by using one patch-reparameterized visual interface across understanding, generation, and editing. Its MoT architecture shares the visual representation and trains these capabilities through task-specific routing and objectives.

  • Unified visual interface: UniSpace performs text-to-image generation, instruction-based image editing, and image understanding in one patch-reparameterized visual representation space.It uses a PR-Qwen-ViT encoder–decoder as the sole visual interface.
  • Unified MoT architecture: The MoT backbone routes text and conditioning-image tokens to an understanding expert, while noised visual tokens are routed to a generation expert.Both experts operate on a common multimodal sequence.
  • Unified visual interface: Unlike prior MoT-based models, UniSpace requires neither a separate VAE encoder nor an additional VAE latent space.Reference, target, and generated images are represented through the same unified visual tokenizer.
  • Unified training objectives: Understanding uses cross-entropy response prediction, whereas generation and editing use balanced flow matching in the shared visual representation space.Generation and editing differ only in their conditioning context.
  • Training details: UniSpace is trained progressively from 256 to 512 and 1024 resolution across three stages, processing approximately 510M instances and 470B multimodal tokens.The visual tokenizer and encoder remain frozen throughout training.
  • Training details: Packed multimodal training concatenates independent examples until a token budget is reached, so task ratios describe expected full-run frequencies rather than every packed sequence.This accommodates heterogeneous multimodal sequence lengths.

5.1 UNIFIED TOKENIZER EVALUATION

Patch Reparameterization equips pretrained semantic ViTs with reconstruction-aware tokens while retaining semantic understanding, yielding strong reconstruction and a favorable reconstruction–generation trade-off.

  • High-Fidelity Image Reconstruction: PR-DINOv2 achieves an rFID of 0.14, a PSNR of 30.84, and an SSIM of 0.90 on ImageNet-1K reconstruction.These results are reported across comparisons involving PR-SigLIP2, PR-DINOv2, and PR-Qwen-ViT.
  • High-Fidelity Image Reconstruction: PR-DINOv2 reduces rFID from 0.57 to 0.14 against matched-backbone RAE, while PSNR rises from 18.86 to 30.84 and SSIM from 0.48 to 0.90.The pretrained ViT backbone remains frozen.
  • High-Fidelity Image Reconstruction: PR-DINOv2 reduces rFID from 0.28 to 0.14 compared with VA-VAE and also surpasses larger generative VAEs in rFID.PR-Qwen-ViT shows that the method generalizes beyond ImageNet-pretrained encoders while retaining semantic capabilities.
  • Multimodal Understanding: PR-SigLIP2 reaches an average understanding score of 64.37 versus 63.39 for SigLIP2-B, while PR-Qwen-ViT reaches 68.94 versus 68.29 for Qwen-ViT.The comparisons use matched language models, projectors, training data, and protocols.
  • ImageNet Generation: PR-DINOv2 reaches an rFID of 0.14 with gFIDs of 2.10 without classifier-free guidance and 1.87 with classifier-free guidance at scale 1.2.RAE, RAEv2, and VTP-L achieve stronger generation results but substantially higher tokenizer rFID values.
  • ImageNet Generation: An entangled semantic–reconstruction representation achieves good reconstruction on real latents but fails after generation, with FID 120.9 using the reconstruction decoder versus 8.07 using the semantic decoder.Approximately 95% of merger-output variation is explained by the semantic pathway.

5.2 SCALING UNIFIED MULTIMODAL MODELING

UniSpace scales the unified visual representation into an 8B multimodal model for image editing and text-to-image generation. It achieves strong benchmark performance while preserving an understanding pathway, whose system-level scores are not the paper’s primary evidence.

  • Image Editing: UniSpace achieves an ImgEdit overall score of 4.28, exceeding SenseNova-U1 at 3.90 and BAGEL at 3.20 while approaching 32B Emu3.5 at 4.41.The result demonstrates strong editing capability at the 8B scale.
  • Image Editing: UniSpace obtains GEdit overall scores of 7.41 on English and 7.38 on Chinese, yielding a bilingual average of 7.39.It outperforms BAGEL and remains close to dense SenseNova-U1 8B.
  • Image Editing: Qualitative editing comparisons cover object addition, style transfer, scene replacement, and object substitution, with UniSpace preserving unrelated regions while following instructions.The comparison includes BAGEL and SenseNova-U1.
  • Text-to-Image Generation: UniSpace achieves an overall GenEval score of 0.84, performing well on object, positional, and attribute-binding tasks while remaining weaker on counting and color binding.GenEval evaluates object-centric compositional alignment.
  • Text-to-Image Generation: UniSpace reaches a bilingual OneIG-Bench average of 0.547, slightly exceeding Emu3.5 at 0.546 and SenseNova-U1 at 0.542.It also obtains the highest style scores among compared methods on both language subsets.
  • Text-to-Image Generation: UniSpace achieves a DPG-Bench score of 86.49 and the highest relation score among compared models at 94.97.Its entity score is 92.26, while global and attribute scores remain lower than some alternatives.
  • Unified Capability Evaluation: UniSpace’s understanding pathway is retained, but controlled unified-representation experiments rather than system-level understanding scores provide the main evidence.The system is optimized primarily for generation and editing, not standard VLM benchmarks.

6 LIMITATION

UniSpace demonstrates that a shared visual representation is viable, but its gains are not uniform across downstream tasks. The system prioritizes generation and editing, while visual understanding remains constrained by architectural trade-offs.

  • UniSpace is primarily optimized for generation and editing, and its system-level understanding performance still lags behind dedicated vision–language models.
  • The shared representation cannot be freely updated on understanding data without potentially compromising generative capabilities.
  • This architectural constraint limits further improvements in visual understanding.

7 CONCLUSION

The paper presents Patch Reparameterization as a minimal way to turn a pretrained semantic ViT into a unified visual tokenizer. Scaling this representation to UniSpace shows that one visual space can support understanding, editing, and text-to-image generation at 8B scale.

  • Patch Reparameterization adds a reconstruction-aware patch pathway while retaining the original semantic pathway and frozen Transformer blocks.The resulting representation supports understanding, reconstruction, and generation within one frozen parameter space.
  • A single visual space can replace the conventional ViT–VAE dual interface in a large multimodal system.UniSpace scales the same frozen representation into an 8B mixture-of-experts model.
  • UniSpace supports multimodal understanding, high-quality image editing, and practical text-to-image generation at compact 8B model scale.The system achieves competitive text-to-image generation while maintaining high-quality image editing.

A ABLATION ON BALANCED FLOW MATCHING

The balanced flow-matching ablation finds that reconstruction supervision has a middle optimum rather than benefiting monotonically from larger weighting. The final model therefore uses sufficient but not overwhelming reconstruction emphasis.

  • The ablation isolates the effect of semantic–reconstruction balance by keeping the encoder, decoder, schedule, representation, and sampling setting fixed.
  • 0.75 reconstruction weight gives the best FID, improving from 8.99 at λr = 0.25 to 7.10, while λr = 0.92 yields 7.39.All variants use the same 768+128 representation and controlled training and sampling settings.
  • Increasing reconstruction weight beyond λr = 0.75 does not continue improving generation.The sweep supports a middle optimum rather than a monotonic preference for larger reconstruction weights.

B ABLATION ON RECONSTRUCTION COMPRESSION

Reconstruction compression trades some pure reconstruction quality for substantially easier generative modeling. Reducing the reconstruction stream from 768 to 128 channels preserves a more favorable generation–reconstruction balance.

  • Increasing reconstruction channels from 128 to 768 improves rFID from 0.163 to 0.085 and PSNR from 29.79 to 33.61.The comparison keeps the semantic component and factorized concatenation design unchanged.
  • Full-dimensional reconstruction worsens FID from 9.95 to 15.51 at epoch 20 and from 6.92 to 11.07 at epoch 40.The larger reconstruction stream is substantially harder for the DiT prior to model.
  • Reducing the reconstruction stream from 768 to 128 channels slightly weakens pure reconstruction metrics but substantially improves generative modeling under the same early training budget.

C ENTANGLED REPRESENTATION AND DIAGNOSTIC PROTOCOL

The diagnostic protocol tests whether an entangled merger can preserve and generate both semantic and reconstruction information. Its results show semantic variation dominates the merged space, making reconstruction-relevant directions difficult for a generative model to identify and control.

  • Diagnostic protocol: The protocol merges semantic and reconstruction-oriented encoder outputs with an MLP, while aligning the merger with both semantic and reconstruction objectives.The semantic decoder and high-fidelity reconstruction decoder are pretrained separately, with their corresponding encoders frozen during the diagnostic experiment.
  • Generative diagnostic: The merged representation does not preserve an explicit semantic–reconstruction decomposition, unlike the factorized representation used in the final method.This entanglement makes reconstruction information present but difficult for the generative model to identify and control.
  • Generative diagnostic: A DiT prior trained on the merged representation produces FID 120.9 with the reconstruction decoder versus FID 8.07 with the semantic decoder.The decoder-dependent gap indicates that the prior primarily learns semantic-dominant variation in the entangled representation.
  • Variance analysis: ρs ≈95% of merger-output variation is explained by the semantic pathway over the evaluation image set.Reconstruction-relevant directions occupy a relatively small and difficult-to-control part of the entangled latent space, motivating explicit factorization.
Loading 2608.08676v1…