UniLiP: Adapting CLIP for Unified Multimodal Understanding, Generation and Editing

1 INTRODUCTION

• Prior CLIP-based unified methods either degrade understanding through feature quantization or produce inconsistent reconstructions because CLIP lacks pixel details.These limitations create a trade-off between semantic comprehension and reconstruction quality.Source: §1 · pp. 1–2
• UniLIP trains a pixel decoder against frozen CLIP first, then trains CLIP with self-distillation constraints to preserve its feature distribution and comprehension.The method uses the frozen CLIP as teacher during the second stage.Source: §1 · pp. 2–3
• UniLIP uses multimodal hidden states and learnable query embeddings as dual diffusion conditions, combining visual detail with MLLM reasoning for editing consistency.The design addresses the information loss caused by fixed-length queries alone.Source: §1 · pp. 2–3
• UniLIP, trained with 40M public images, outperforms all similarly-sized unified models on GenEval and WISE with scores 0.90 and 0.63, and significantly surpasses larger UniWorld-V1 and BAGEL on ImgEdit with 3.94.Source: §1 · p. 2
• UniLIP adapts CLIP for understanding, reconstruction, generation, and editing without losing its original understanding capabilities.The framework extends CLIP beyond semantic encoding to support reconstructable visual representations.Source: §1 · p. 3
• Frozen CLIP reconstructions omit image details such as edge objects and letters, whereas UniLIP captures them and answers the associated questions correctly.The qualitative comparison links reconstruction detail with understanding outcomes.Source: Figure 2 · p. 3

MLLMs for Image Understanding

• CLIP-based visual encoders remain strong for multimodal understanding but generally lack generative capabilities.Subsequent work improves CLIP through model design, scale, training, and fine-grained perception extensions.Source: §2 · p. 3

Unified Visual Tokenizers

• Unified visual tokenizers based on VQ/VAE or quantized CLIP features trade semantic quality or reconstruction detail for unification.UniLIP instead aims to maintain CLIP understanding while achieving excellent reconstruction.Source: §2 · p. 3

Architectures for Unified MultiModal Understanding and Generation

• Unified multimodal architectures evolved from shared weights to MoE and query-based bridges, while UniLIP adds dual conditioning to address editing limitations.Prior query-based methods can lose information when fixed-length queries represent reference images.Source: §2 · p. 3

Instruction-Guided Image Editing

• Instruction-guided editing methods use MLLM semantic embeddings, but CLIP-based systems often still require VAE features because CLIP lacks pixel details.UniLIP adapts CLIP for reconstruction to support consistent editing without resolution constraints or VAE dependence.Source: §2 · p. 4

3.1 FINETUNING CLIP FOR IMAGE RECONSTRUCTION

• A two-stage reconstruction scheme first trains a pixel decoder with frozen CLIP features, then fine-tunes CLIP using self-distillation to improve reconstruction without losing original capabilities.Stage 1 uses reconstruction and perceptual losses; Stage 2 constrains feature drift from the original CLIP representation.Source: §3.1 · pp. 4–5, Figure 3 · p. 4
• UniLIP reconstructs images as a CLIP–projection–pixel-decoder pipeline, with CLIP features aligned to the decoder input before pixel reconstruction.The projection is implemented with several MLPs, while the pixel decoder produces the reconstructed image.Source: §3.1 · p. 4
• The self-distillation objective penalizes differences between original and fine-tuned CLIP features, with λ set to 1 and CLIP trained at 0.1 times the global learning rate.These settings are reported to help preserve original abilities.Source: §3.1 · p. 5

3.2 DUAL CONDITIONS FOR IMAGE GENERATION AND EDITING

• UniLIP’s continuous features support generation and editing through query-based diffusion conditioning, while dual conditions address the information bottleneck caused by fixed-length query embeddings.Fixed query tokens can lose reference-image details in editing; multimodal hidden states supplement this missing information.Source: §3.2 · pp. 5–6
• A dual-condition architecture feeds both multimodal hidden states and learnable query embeddings to the diffusion transformer, supplementing query information with reference-image details.For generation, the reference-image input is removed; the architecture uses query embeddings for MLLM reasoning and hidden states for richer context.Source: Figure 4 · p. 5, §3.2 · p. 6

3.3 TRAINING SCHEMES

• The unified pipeline trains a connector first with frozen MLLM and diffusion transformer, then jointly trains connector and diffusion transformer for generation and editing while keeping the MLLM frozen.A final supervised fine-tuning stage uses high-quality instruction data to improve fidelity and prompt alignment.Source: §3.3 · p. 6

4.2 COMPARISON WITH STATE-OF-THE-ARTS

• UniLIP improves reconstruction over prior CLIP-based tokenizers by 0.58 rFID at 256×256 and over Emu2 by 2.96 rFID and 11.13 PSNR at 448×448.Its dedicated reconstruction training better preserves pixel-level details from CLIP features.Source: §4.2 · p. 8
• UniLIP achieves state-of-the-art visual understanding among similarly scaled unified models and surpasses larger CLIP-quantizing methods.The evaluated benchmarks include MME-P, MMB, MMMU, MM-Vet, SEED, AI2D, and MMVP.Source: §4.2 · p. 8
• 0.90 on GenEval, 0.63 on WISE, and 3.94 on ImgEdit demonstrate strong generation and editing performance.UniLIP surpasses or compares favorably with larger unified models on these benchmarks.Source: §4.2 · p. 8

4.3 ABLATION STUDY

• Direct reconstruction finetuning gives the best PSNR but causes severe understanding degradation, whereas the adopted strategies preserve comprehension with only minor reconstruction loss.Removing self-distillation decreases MMBench by 54.2 points.Source: §4.3 · p. 9, Table 6 · p. 10
• The two-stage method reduces the initial distillation-loss spike from 0.0939 to 0.0497 and needs 3× fewer convergence iterations than single-stage training.Stage 1 pre-aligns the pixel decoder and avoids gradient instability from initial misalignment.Source: §4.3 · p. 9, Figure 6 · p. 10
• The dual-condition architecture combines multimodal hidden states for reference-image detail with query embeddings for MLLM reasoning, improving both WISE and editing consistency.Query embeddings improve WISE by 5 points, while relying on them alone yields only 3.38 on ImgEdit.Source: §4.3 · p. 9
• Replacing UniLIP’s reference encoder with CLIP lowers editing performance from 3.81 to 3.42, while replacing the target encoder with VAE lowers WISE from 0.56 to 0.48.The ablations attribute these drops to missing pixel detail and reduced generation or editing quality.Source: §4.3 · p. 10

• UniLIP extends CLIP to support understanding, reconstruction, generation, and editing simultaneously.Source: §5 · p. 10
• Two-stage training with self-distillation addresses the trade-off between semantic comprehension and pixel-level detail preservation.Source: §5 · p. 10
• The dual-condition architecture connects MLLMs and diffusion models for image synthesis and editing with improved detail and consistency.Source: §5 · p. 10

REPRODUCIBILITY STATEMENT

• The authors state that model configurations, datasets, hardware, and training and inference hyperparameters are documented, with pretrained models and datasets open-sourced.Source: REPRODUCIBILITY STATEMENT · p. 11

A.1 ARCHITECTURE

• UniLIP uses a 32× pixel decoder after 28× image downsampling and projects high-dimensional features to 32 decoder dimensions.Source: §A.1 · p. 16

A.2 TASK BALANCE

• Training samples are rebalanced so generation-to-editing ratios are approximately 2:1 during joint training and 1:1 during instruction tuning.Source: §A.2 · p. 16

A.3 CLASSIFIER FREE GUIDANCE

• Classifier-free guidance nullifies text prompts with probability 0.1, retains editing reference images, and uses guidance scale 4.5.Source: §A.3 · p. 16

• Stage 2 introduces a discriminator after 200k steps to improve reconstruction fidelity.Source: §A.4 · p. 16
• The Stage 2 objective combines MSE, LPIPS, distillation, and weighted GAN losses, with λG set to 0.1.Source: §A.4 · p. 16

B.1 IMAGE RECONSTRUCTION

• 0.35 rFID, 25.02 PSNR, and 0.790 SSIM demonstrate strong reconstruction performance for the 3B UniLIP model.Source: §B.1 · p. 16

B.2 IMAGE GENERATION AND EDITING

• The reported evaluations cover GenEval, WISE, and ImgEdit, with full results provided in the cited tables.Source: §B.2 · p. 16

B.3 INFERENCE SPEED

• UniLIP generates images substantially faster than similarly sized unified models by using fewer iterations and a lightweight pixel decoder instead of BLIP3-o’s two-stage diffusion process.Source: §B.3 · pp. 16–17, Table 9 · p. 17

B.4 REASONING EDITING

• The 3B model significantly outperforms larger Step1X-Edit and Bagel models on the RISE benchmark, supporting strong reasoning-based editing.Source: §B.4 · p. 17

B.5 LINEAR AND ATTENTIVE PROBE

• UniLIP achieves ImageNet classification performance comparable to the original InternViT under both linear and attentive probing.Source: §B.5 · p. 17

C.1 RECONSTRUCTION TRAINING CURVE

• After an initial rise, distillation loss decreases while understanding recovers and rFID improves, indicating that UniLIP balances reconstruction with preserved comprehension.Source: §C.1 · p. 17, Figure 7 · p. 17

C.2 GENERATION AND EDITING TRAINING STAGE

• All training stages are necessary: instruction tuning improves GenEval, the second stage affects WISE and ImgEdit, and alignment protects pretrained capabilities.Source: §C.2 · pp. 17–18, Table 11 · p. 18

C.3 DISTILLATION LOSS

• MSE with weight 1.0 provides the best understanding-preservation trade-off, whereas classification loss is coarse and excessive weighting can degrade generalization and reconstruction.Source: §C.3 · p. 18

C.4 UNDERSTANDING THE QUERY EMBEDDINGS

• Learnable query embeddings activate internal knowledge for underspecified prompts, enabling details such as “turkey” to guide generation.Source: §C.4 · p. 18, Figure 8 · p. 19

C.5 QUERY LENGTH

• WISE performance improves with query length and converges around L = 256, consistent with a larger textual-reasoning budget.Source: §C.5 · p. 18, Table 14 · p. 18

C.6 JOINT TRAINING

• Joint generation-editing training significantly outperforms editing-only training across metrics because large-scale generation data supplies a more robust image distribution.Source: §C.6 · p. 20, Table 15 · p. 20

C.7 COMPARISON WITH DUAL-ENCODER METHODS

• UniLIP-SigLIP achieves lower rFID than SDXL-VAE while maintaining comparable PSNR and SSIM and preserving or slightly improving multimodal understanding.Source: §C.7 · p. 20
• +2.0 GenEval, +5.0 WISE, and +0.14 ImgEdit show UniLIP-JanusFlow outperforming the dual-encoder baseline with half as many image tokens.Source: Table 16 · p. 20, Table 17 · p. 21

D.1 IMAGE RECONSTRUCTION

• UniLIP preserves continuous CLIP features and matches DC-AE reconstruction quality, avoiding the blur and consistency losses observed in competing approaches.Source: §D.1 · pp. 20–21

D.2 IMAGE GENERATION

• Qualitative generation comparisons show stronger positional accuracy, clearer faces, semantic reasoning, and stylistic fidelity than competing models.Source: §D.2 · p. 21

D.3 IMAGE EDITING

• Qualitative editing comparisons show better instruction adherence, object preservation, contextual inpainting, and reduced blurring artifacts.Source: §D.3 · p. 21

D.4 ABLATION OF REFERENCE IMAGE ENCODER

• UniLIP preserves subject identity, texture, and background more faithfully than CLIP during editing.Source: §D.4 · p. 21
• CLIP can follow editing instructions but struggles to preserve the edited subject’s identity, texture, and background.Source: §D.4 · p. 21
• The comparison attributes UniLIP’s greater editing fidelity to its reconstruction training.Source: §D.4 · p. 21

E LIMITATION AND FUTURE WORK

• 3B parameters and 448 ×448 resolution limit UniLIP and contribute to subpar text rendering and editing performance.The authors plan to scale model size and resolution to improve text-rendering-related tasks.Source: E LIMITATION AND FUTURE WORK · p. 21

F THE USE OF LARGE LANGUAGE MODELS

• The manuscript reports that LLMs were used only for language polishing, grammar correction, and readability improvements.The authors state that ideation, data analysis, and conclusions were conducted by the authors.Source: F THE USE OF LARGE LANGUAGE MODELS · p. 21
• The section provides qualitative comparisons for reconstruction, text-to-image generation, image editing, and CLIP versus UniLIP reference encoding.It also identifies GenEval, WISE, and ImgEdit evaluation tables and gives example editing prompts.Source: Figure 9 · p. 22, Figure 10 · p. 23, Figure 11 · p. 24