Source-linked AI summary
OpenVision 3: A Family of Unified Visual Encoder for Both Understanding and Generation
Letian Zhang, Sucheng Ren, Yanqing Liu, Xianhang Li, Zeyu Wang, Yuyin Zhou, Huaxiu Yao, Zeyu Zheng, Weili Nie, Guilin Liu, Zhiding Yu, Cihang Xie
TL;DR
Unified multimodal systems need visual representations that support both semantic understanding and image generation without relying on separate or discretized tokenizers. OpenVision 3 uses a VAE-plus-ViT encoder with reconstruction, contrastive, and captioning objectives, and frozen-encoder evaluations show strong generation and competitive understanding. The method also indicates that its shared representation can support mutually beneficial learning across the two regimes.
Problem
Separate or quantized visual tokenizers complicate unified modeling and can limit synergy or generation quality.
Method
OpenVision 3 feeds VAE latents into a ViT and jointly trains reconstruction with contrastive and captioning objectives.
Results
OpenVision 3 outperforms other unified tokenizers in reconstruction and generation while matching or exceeding CLIP on multimodal understanding benchmarks.
Takeaways & Limitations
A single VAE–ViT representation can support both generative and semantic downstream tasks, with evidence of mutual promotion between understanding and generation.
Takeaways & Limitations
The training objective uses an understanding-loss weight twice as large as the reconstruction-loss weight to preserve generative quality without impairing understanding.
Abstract
from arXiv · showhide
This paper presents a family of advanced vision encoder, named OpenVision 3, that learns a single, unified visual representation that can serve both image understanding and image generation. Our core architecture is simple: we feed VAE-compressed image latents to a ViT encoder and train its output to support two complementary roles. First, the encoder output is passed to the ViT-VAE decoder to reconstruct the original image, encouraging the representation to capture generative structure. Second, the same representation is optimized with contrastive learning and image-captioning objectives, strengthening semantic features. By jointly optimizing reconstruction- and semantics-driven signals in a shared latent space, the encoder learns representations that synergize and generalize well across both regimes. We validate this unified design through extensive downstream evaluations with the encoder frozen. For generation, we test it under the RAE framework: ours substantially surpasses the standard CLIP-based encoder (e.g., gFID: 1.87 vs. 2.54 on ImageNet). For multimodal understanding, we plug the encoder into the LLaVA-1.5 and LLaVA-NeXT framework: it performs comparably with a standard CLIP vision encoder (e.g., 63.3 vs. 61.2 on SeedBench, and 59.2 vs. 58.1 on GQA). We provide empirical evidence that generation and understanding are mutually beneficial in our architecture, while further underscoring the critical role of the VAE latent space. We hope this work can spur future research on unified modeling.
1 Introduction
OpenVision 3 addresses the difficulty of using one visual representation for both understanding and generation. It combines VAE-based reconstruction with semantic supervision in a shared encoder and evaluates transfer with the encoder frozen.
- Separate visual tokenizers encode images into semantic and pixel-reconstructable tokens, increasing system complexity and potentially limiting synergy.
- Quantized shared tokenizers introduce discretization errors that can limit generation quality, leaving continuous unified visual tokenization open.
- OpenVision 3 stacks a ViT encoder on a pretrained VAE encoder to construct a unified visual tokenizer.
- The generation branch reconstructs images, while contrastive and captioning objectives in the understanding branch strengthen semantic supervision.
- Frozen-tokenizer evaluations test OpenVision 3 in multimodal understanding, reconstruction, and ImageNet generation settings.
2 Related Work
Prior unified multimodal systems commonly separate semantic and generative visual features or fuse them through discrete tokenizers. This work sits within efforts to build a single representation spanning both roles.
- Vision-language pretraining has developed from contrastive learning toward captioning, sigmoid losses, self-distillation, masked prediction, and multi-teacher distillation.
- Unified multimodal systems often extract generation-oriented and understanding-oriented features separately before combining them.
- BAGEL uses FLUX-VAE for low-level features and SigLIP2 for semantic features, while UniWorld-V1 concatenates separately computed feature types.
- Early unified tokenizers predominantly use discrete tokens and vector quantization to build shared codebooks for semantic and pixel-level features.
3 Method
OpenVision 3 constructs unified visual features by processing VAE latents with a ViT and decoding the result through separate reconstruction and understanding branches. Its objective jointly weights reconstruction and semantic losses, with progressive training from low to high resolution.
- OpenVision 3: A unified tokenizer: The method first encodes images into VAE latents, then applies a ViT encoder to obtain unified representations for understanding and generation.
- OpenVision 3: A unified tokenizer: The unified feature enters separate reconstruction and understanding branches with distinct decoding architectures.
- OpenVision 3: A unified tokenizer: The reconstruction branch adds Gaussian noise, maps perturbed features back to VAE latents, and decodes them into reconstructed images.
- OpenVision 3: A unified tokenizer: The understanding branch combines autoregressive captioning with contrastive alignment between unified visual and text features.
- OpenVision 3: A unified tokenizer: The overall objective combines reconstruction and understanding losses, with the understanding weight set to twice the reconstruction weight.
- Training settings: Progressive training moves from 128×128 pretraining to 224×224 or 256×256 finetuning, keeping the two stages at roughly a 10:1 epoch ratio.
- Training settings: The VAE remains frozen during training, while the ViT, decoders, text modules, and linear layer remain trainable.
4 Experiments
OpenVision 3 is evaluated as a unified tokenizer for reconstruction, generation, and multimodal understanding. Across these settings, it preserves fine-grained visual information, achieves strong ImageNet generation metrics, and matches or exceeds CLIP-based understanding performance.
- Evaluation setup: OpenVision 3 is evaluated for reconstruction, ImageNet generation, and multimodal understanding using RAE and LLaVA frameworks.Generation uses DiT with a wide DDT head, while understanding is tested under LLaVA-1.5 and LLaVA-NeXT.
- Reconstruction performance: OpenVision 3 achieves PSNR 30.92 dB versus UniTok’s 25.34 dB and Vila-U’s 22.24 dB, while reaching LPIPS 0.053 versus UniTok’s 0.132.On COCO, its rFID is 1.601 versus UniTok’s 3.918.
- Reconstruction performance: OpenVision 3 preserves textual content and intricate details more faithfully than SD-VAE and RAE in qualitative reconstructions.Reported examples include recovering hull characters and capturing flower, foliage, and tiger-eye details.
- Generation performance: OpenVision 3 records gFID 1.87 versus 2.06 for SD-VAE with SiT and 2.54 for CLIP under RAE.The tokenizer is reported to outperform the compared tokenizers across gFID, IS, precision, and recall.
- Understanding performance: Under LLaVA-1.5 and LLaVA-NeXT, OpenVision 3 matches or exceeds CLIP across multimodal benchmarks.In LLaVA-NeXT, SeedBench reaches 63.3 versus 61.2 and GQA reaches 59.2 versus 58.1 against CLIP.
5 Discussion
OpenVision 3’s reconstruction and understanding objectives exhibit reciprocal benefits, while VAE latents improve unified performance without compromising understanding. Decoder and encoder scaling also reveal different trade-offs across reconstruction, generation, and multimodal understanding.
- 5.1 Reciprocal synergy between understanding and reconstruction: Semantic supervision lowers reconstruction loss even without reconstruction objectives, while reconstruction loss does not significantly change caption or contrastive losses.These observations support a mutually beneficial relationship between the two branches.
- 5.1 Reciprocal synergy between understanding and reconstruction: Adding semantic loss improves reconstruction performance, while reconstruction-only training leaves contrastive loss nearly stagnant and caption loss marginally declining.The reconstruction task intrinsically facilitates generative semantic tasks, but the reverse benefit is stronger for reconstruction.
- 5.2 Effectiveness of VAE latent space: 1.23 gFID: OpenVision 3’s VAE-latent tokenizer outperforms its non-VAE counterpart in generation.The comparison is reported under the VAE-latent and non-VAE variants.
- 5.2 Effectiveness of VAE latent space: OpenVision 3 achieves superior reconstruction quality in rFID with VAE latents while remaining competitive on PSNR, SSIM, and LPIPS.The reported advantage is strongest for rFID; the other metrics show only slight differences.
- 5.2 Effectiveness of VAE latent space: OpenVision 3 consistently outperforms its non-VAE counterpart in multimodal understanding under identical settings, including a lead across all LLaVA-NeXT metrics.Under LLaVA-1.5, it wins four of six metrics.
- 5.3 The selection of ViT decoder size: Base-sized ViT decoders reduce reconstruction loss while maintaining training stability, whereas Large-sized decoders cause significant instability and decoder size has negligible understanding impact.The Base-sized decoder is presented as the best balance of performance and computational efficiency; M/1, B/1, and L/1 training durations are 6.1, 6.0, and 6.9 hours.
- 5.4 Scaling behavior of ViT encoder: B-size and L-size encoders yield comparable reconstruction and generation results, but scaling to L-size provides substantial gains in multimodal understanding.The scaling comparison separates generative performance from understanding performance.
6 Conclusion
OpenVision 3 introduces a unified VAE–ViT vision encoder whose shared representation supports understanding and generation through joint reconstruction- and semantics-driven training. Evaluations report superior reconstruction and generation, competitive semantic performance with CLIP, and mutual promotion between the two capabilities.
- 6 Conclusion: OpenVision 3 couples a VAE with a ViT to generate one unified representation for downstream understanding and generation tasks.Its tokenizer is trained jointly with reconstruction- and semantics-driven signals.
- 6 Conclusion: OpenVision 3 outperforms other unified tokenizers in reconstruction and generation while remaining competitive with CLIP on semantic tasks.The conclusion summarizes results across generative and understanding evaluations.
- 6 Conclusion: The architecture enables a mutual promotion relationship between image understanding and generation.The paper reports this relationship as an empirical finding of the unified design.