PaperlayerView exact revision on arXiv

Source-linked AI summary

Unifying Contrastive and Generative Objectives for Visual Understanding and Text-to-Image Generation

Chao Li, Tianhong Li, Sai Vidyaranya Nuthalapati, Hong-You Chen, Satya Narayan Shukla, Jianpeng Cheng, Yonghuan Yang, Jun Xiao, Xiangjun Fan, Aashu Singh, Dina Katabi, Shlok Kumar Mishra

arXiv:2603.02667v2cs.CVcs.LG

TL;DR

DREAM addresses the difficulty of jointly training text-image contrastive alignment and text-to-image generation, whose masking requirements conflict. It uses Masking Warmup to expose the encoder to both regimes and enables Semantically Aligned Decoding for partial-generation selection. Across discriminative and generative benchmarks, DREAM improves over CLIP and FLUID, supporting synergy between the objectives.

  • Problem

    Contrastive alignment needs near-complete visible tokens while masked generative modeling needs heavy corruption, making end-to-end joint optimization difficult.

  • Method

    DREAM shifts the center of a wide masking distribution during training so low and high masking ratios coexist, then uses jointly learned visual-text alignment to select among partially decoded image candidates.

  • Results

    DREAM improves discriminative benchmarks over CLIP and achieves 6.2% lower T2I FID than FLUID on CC12M while maintaining competitive CLIP score.

  • Takeaways & Limitations

    Text-image contrastive and generative objectives are synergistic within a single end-to-end trainable architecture.

  • Takeaways & Limitations

    Experiments use CC12M rather than hundreds-of-millions-scale datasets, so verification at that scale remains future work.

Abstract

from arXiv · show

Unifying text-image contrastive learning and text-to-image (T2I) generation in a single end-to-end model is challenging because the two objectives demand opposing masking regimes: contrastive alignment needs near-complete visible tokens, while masked generative modeling needs heavy corruption. We introduce DREAM, a unified framework that resolves this conflict through Masking Warmup, a schedule that shifts the center of the masking distribution over training, so low and high masking ratios coexist at every step. This co-exposure lets a single jointly-trained encoder serve both objectives. The resulting stable optimization unlocks Semantically Aligned Decoding at inference: the text encoder, trained against visual embeddings at all masking ratios, can score partially generated images and select the best trajectory with as little as 12.5% of the image decoded, improving both FID and throughput. DREAM outperforms its single-objective baselines, CLIP and FLUID: on ImageNet linear-probing (+1.1%), 5-shot transfer (+4.1%), ADE20K segmentation (+1.9%), and NYU depth estimation (+6.25%) over CLIP, and on CC12M FID (+6.2%) over FLUID while maintaining CLIP Score. Together, these gains show that text-image contrastive and generative objectives, when properly unified, are synergistic rather than competing.

1 Introduction

DREAM addresses the structural conflict between contrastive alignment and masked generation by scheduling masking so both objectives can be jointly optimized. This unified training also enables semantically aligned decoding and improves discriminative and generative performance.

  • Contrastive alignment needs near-complete visible semantics, whereas masked generation requires heavy corruption; naive joint training collapses linear probing to 4.6% and fails to converge on generation.
  • Masking Warmup shifts the masking distribution from low toward high ratios while retaining both regimes throughout training.The schedule lets the encoder establish text-aligned representations before the decoder receives heavier corruption.
  • Semantically Aligned Decoding scores partially generated candidates with the model’s text-image alignment, selecting a trajectory after as little as 12.5% of the image is decoded.The approach improves generation FID by 7.0% and is enabled by embeddings learned across visibility levels.
  • +1.1% linear probing on ImageNet-1K, +4.1% 5-way 5-shot transfer, +1.9% ADE20K, and 6.25% lower RMSE on NYU Depth v2 are reported over CLIP.
  • DREAM improves T2I FID by 6.2% over FLUID on CC12M while maintaining competitive CLIP score.
  • The authors characterize contrastive and generative objectives as mutually reinforcing within a single end-to-end architecture.

2 Related Work

Prior work separates generation from text-image contrastive alignment through objective choices, multiple stages, or frozen external encoders. DREAM instead unifies the relevant objectives in one jointly trained architecture and introduces self-guided partial-generation selection.

  • Some multimodal models unify generation across modalities without a text-image contrastive objective, so the masking conflict addressed by DREAM does not arise.
  • Multi-stage systems avoid co-optimizing representation and generation by splitting training into multiple stages, with Janus also using separate understanding and generation encoders.
  • MAGE and ST-AR use intra-modal image contrast rather than text-image contrast, so they do not encounter DREAM’s text-image masking conflict.
  • REPA and RAE align generation features to frozen external vision encoders rather than a jointly trained text encoder.
  • DREAM’s Semantically Aligned Decoding selects among partial latent generations using its own learned alignment instead of fully decoding candidates for external CLIP reranking.

3 Implementation Details

Experiments use CC12M with a controlled training and evaluation setup. DREAM adds a progressive masking warmup to otherwise specified optimization procedures.

  • All experiments train on CC12M, which contains approximately 11.3M text-image pairs.Images are center-cropped to 256 × 256 and horizontally flipped for augmentation.
  • Models train for 49 epochs with AdamW, batch size 2048, and a learning rate warmed linearly for 12 epochs to 8 × 10^-4.
  • DREAM uses a 36-epoch progressive masking warmup in addition to the shared training procedure.
  • The masking distribution’s mean increases linearly from 0 to 1.0 over the first 36 epochs and then remains fixed at 1.0.

4 Method

DREAM combines a ViT-based encoder-decoder over continuous image latents with contrastive alignment and diffusion reconstruction. At inference, it uses the encoder to select the most text-aligned among partially decoded candidates.

  • Architecture: DREAM uses a ViT-based encoder-decoder over continuous image latents, applying text conditioning only in the decoder.This design targets language-aligned visual features without relying on language shortcuts in the encoder.
  • Architecture: Images are represented as continuous latent tokens from a pretrained Stable Diffusion VAE, preserving fine-grained spatial information with computational efficiency.
  • Architecture: The vision encoder processes buffer tokens and unmasked image tokens, following the MAR architecture.
  • Architecture: DREAM uses distinct text encoders for contrastive alignment and generation, including a CLIP text transformer and a frozen T5-XXL encoder.
  • Architecture: The decoder predicts masked continuous tokens through bidirectional attention, using self-attention over visual tokens and text-conditioned cross-attention.
  • Diffusion reconstruction: The diffusion head uses a cosine noise schedule with 1000 training steps and 100 resampled inference steps.
  • Training objectives: Diffusion loss is computed when more than 50% of tokens are masked, reflecting the high masking ratios required for generative modeling.
  • Training objectives: The contrastive objective uses symmetric image-to-text and text-to-image InfoNCE terms averaged as Lclip = (LI + LT)/2.

5 Experiments

Experiments evaluate DREAM’s masking schedule, complementary reconstruction objective, unified visual understanding and T2I performance, decoding efficiency, and scaling. Across these evaluations, DREAM improves representation and generation quality while preserving performance under sparse visibility and enabling efficient semantic trajectory selection.

  • DREAM is evaluated on ImageNet-1K representation quality and CC12M text-to-image generation quality, with all models trained for 49 epochs on CC12M.
  • Masking Warmup Ablations: Naive joint training with fixed high masking collapses representation quality because visible tokens carry too little global semantics for effective text alignment.The fixed high-masking variant also produces poor generation, and encoder–decoder convergence fails under sustained high masking.
  • Masking Warmup Ablations: A two-stage CLIP-pretraining baseline similarly collapses regardless of stage-1 duration, supporting the need for sustained contrastive signal during high-masking generative training.
  • Diffusion Reconstruction: 72.5% to 70.8%: removing diffusion reconstruction lowers Linear Probing accuracy below CLIP’s 71.6% baseline.The reconstruction loss supplies spatial supervision when InfoNCE weakens under high masking, while contrastive alignment limits representational drift.
  • Unified Performance: 72.7%: DREAM’s Linear Probing accuracy exceeds CLIP by 1.1% and FLUID by 24.6%, indicating stronger transferable representations.
  • Unified Performance: +4.1%: DREAM exceeds CLIP on average in few-shot image classification.
  • Unified Performance: +1.9% on ADE20K and 6.25% lower RMSE on NYU Depth v2: DREAM improves dense prediction over CLIP.DREAM achieves 36.8% mIoU on ADE20K and matches REPA on NYU Depth v2.
  • Unified Performance: At masking above 0.8, DREAM achieves over 6.2× CLIP’s zero-shot accuracy.Masking Warmup improves robustness for CLIP-M and DREAM, while DREAM consistently outperforms CLIP-M across masking levels.

6 Conclusion

DREAM unifies visual understanding and text-to-image generation by reconciling contrastive and generative objectives with Masking Warmup, then using the resulting joint representations for Semantically Aligned Decoding. The paper reports that these objectives are synergistic within a single end-to-end trainable architecture.

  • DREAM reconciles text-image contrastive and generative objectives through Masking Warmup during training.
  • Semantically Aligned Decoding exploits the resulting joint representations during inference.
  • The results show that the two objectives are synergistic within a single end-to-end trainable architecture.

A.1 Limitations

The evaluation is conducted on the CC12M dataset, so empirical verification at the hundreds-of-millions data scale remains future work. DREAM-L also adds modest training cost and inherits broader risks associated with high-quality text-to-image generation.

  • All experiments use CC12M, approximately 11.3M image-text pairs, to isolate single-stage joint optimization with an unfrozen encoder.
  • Empirical verification at the hundreds-of-millions data scale remains future work.
  • DREAM-L incurs approximately 4% additional training time and GPU memory over FLUID-L, plus an 11.6% parameter increase.
  • DREAM inherits risks including non-consensual or misleading imagery, bias amplification, hallucinated objects, and attribute-binding errors.

B.1 Qualitative Results

Qualitative results show that Semantically Aligned Decoding produces images with more stable structure, clearer details, fewer low-level artifacts, and stronger prompt consistency. The text-guided retrieval step favors partially decoded latents that match intended content while reconstructing cleanly.

  • Semantically Aligned Decoding produces clearer details and more consistent prompt alignment than decoding without it.
  • Images generated with Semantically Aligned Decoding exhibit more stable structure and fewer low-level artifacts.
  • Text-guided retrieval selects intermediate latents that match intended content and reconstruct cleanly.
  • The decoding process favors samples that are semantically coherent with the prompt while maintaining stronger distribution-level realism.

B.2.1 Stability of Masking Warm-up

Masking Warmup supports stable joint optimization when the masking distribution retains sufficient variance, preserving lightly masked samples for contrastive alignment while training generation under heavier corruption. Fixed high masking and several alternative training setups collapse, whereas suitable warmup settings improve both linear probing and FID over training.

  • Stability of Masking Warm-up: With σ=0.45 and σ=0.55, Linear Probing and FID improve monotonically throughout training, while σ=0.35 causes Linear Probing to degrade after warm-up ends.
  • Stability of Masking Warm-up: When σ> 0.35, Linear Probing and FID remain stable with increasing training duration after masking warm-up ends.
  • Stability of Masking Warm-up: A larger masking standard deviation retains more lightly masked samples after warm-up, preserving visual context for the CLIP loss.
  • Stability of Masking Warm-up: The results indicate that masking warm-up enables stable joint optimization when the masking standard deviation is appropriately defined.
  • Stability of Masking Warm-up: Reintroducing the CLIP loss during Stage 2 joint training still causes both Linear Probing and FID to collapse across initialization strengths.
  • Stability of Masking Warm-up: Fixed high masking remains structurally incompatible with effective contrastive alignment regardless of initialization.
  • Downstream Evaluation: +4.1% average 5-way, 5-shot transfer performance across 14 datasets is reported over CLIP.
  • Downstream Evaluation: The dense prediction evaluation covers semantic segmentation on ADE20K and depth estimation on NYU Depth V2 using linear probes.

B.2.7 Zero-shot Robustness.

Variable-ratio masking improves DREAM’s robustness to occlusion, while Semantically Aligned Decoding selects among candidates after only a small fraction of decoding.

  • Zero-shot robustness: When more than 20% of the image is masked, CLIP-M and DREAM exceed CLIP’s zero-shot accuracy by roughly 0.7%.DREAM surpasses CLIP-M across all masking levels, with progressive masking warm-up especially beneficial at moderate masking.
  • Semantically Aligned Decoding: DREAM’s Semantically Aligned Decoding scores multiple partially decoded candidates and continues only the highest-scoring trajectory.Candidates are decoded in parallel until the scoring step, after which one candidate continues.
  • Semantically Aligned Decoding: 12.5% of the image is decoded before candidate selection in the analyzed configuration, using NFE = 128, K = 9, and T = 64.The scoring step is t_s = 8, so selection occurs after 8/64 decoding steps.
  • Generation diversity: SD and external CLIP reranking produce nearly identical diversity scores, 0.119 versus 0.117, both within 0.005 of the no-selection baseline.The diversity analysis used 20 candidates per prompt across 50 prompts and an external CLIP ViT-B/32 encoder.

C Additional Ablations

Additional ablations show that DREAM remains stable across several settings, while moderate masking variance and a 36-epoch warm-up best balance representation and generation quality.

  • C.2 Standard Deviation of Masking: 73.1% Linear Probing with UNI improves representation quality but worsens FID to 4.88, showing a representation-generation trade-off at extreme variance.At σ = 0.35 to 0.45–0.55, Linear Probing rises from 72.0% to 72.7% while FID decreases from 4.74 to 4.57.
  • C.2 Standard Deviation of Masking: Moderate masking variance, σ ∈ [0.45, 0.55], yields the best generative quality while remaining close to optimal representation performance.This range supports the reported synergistic behavior of contrastive alignment and T2I generation.
  • C.3 Masking Warm-up Duration: A 36-epoch masking warm-up achieves the lowest FID, 4.57, while Linear Probing remains largely stable across 30–42 epochs.Warm-up duration primarily affects generation fidelity rather than discriminative performance.
  • C.4 Minimum Masking Ratio: Setting γ = 0.5 for diffusion-loss masking produces the lowest tested FID, 4.57, whereas higher γ reduces Linear Probing accuracy.The diffusion objective benefits from sufficiently high masking ratios, but very high minimum ratios harm representation quality.
  • C.5 Maximum Masking Ratio: Increasing the CLIP-loss maximum masking ratio ϕ from 0.25 to 0.75 consistently improves Linear Probing accuracy.A larger upper bound allows more samples to retain enough visible content for contrastive alignment.
  • Masking overlap: The minimum diffusion-loss ratio and maximum CLIP-loss ratio jointly control overlap between samples used by the two objectives.Their interaction influences the balance between generative reconstruction and discriminative alignment.
  • C.6 CLIP Loss Weight: Varying the CLIP-loss weight leaves Linear Probing and FID largely unchanged, supporting the framework’s stability.The ablation evaluates the coefficient balancing representation and generation objectives.
  • C.7 CLIP Loss Application: Applying CLIP loss only to buffer tokens slightly reduces Linear Probing while leaving FID unchanged.The comparison tests supervision on buffer tokens versus both buffer and image tokens.

D.6 Training Efficiency

DREAM-L adds modest training overhead relative to FLUID-L despite incorporating two text encoders, with measurements taken on identical NVIDIA A100 hardware.

  • Training overhead: DREAM-L increases parameters by 11.6% and training compute by 13.5% over FLUID-L.The added parameters arise from the CLIP text encoder.
  • Training overhead: DREAM-L requires only 4.2% more training time and 4.2% additional GPU memory than FLUID-L.The comparison uses NVIDIA A100 GPUs and the same hardware setup for training-overhead and memory measurements.
  • Practical cost: The reported figures indicate that DREAM’s additional architectural components impose no significant practical training burden.This conclusion accompanies the modest time and memory increases relative to the baseline.

F Training Settings

DREAM uses a ViT encoder–decoder over continuous image latents, combining masked diffusion reconstruction with image–text contrastive learning and optional semantic candidate selection during decoding.

  • Training procedure: DREAM’s training loop samples masking ratios according to a schedule, forwards unmasked tokens through the encoder, reconstructs masked tokens, and backpropagates the combined loss.The loop also includes teacher-feature alignment in the REPA baseline procedure.
  • Training objective: The training objective combines MAR diffusion reconstruction loss with a weighted CLIP image–text loss.The encoder image embedding is mean-pooled before applying the CLIP loss.
  • Inference: Semantically Aligned Decoding generates N candidates until a threshold step, scores them with the text encoder, then continues decoding the best candidate.The selected candidate is unpatchified into the final image after the remaining decoding steps.
  • Generated examples: The reported DREAM-L, DREAM-H, and DREAM-G image examples are generated from CC12M captions with CFG = 5.0.The examples correspond to models with 0.57B, 1.1B, and 2.4B parameters, respectively.
Loading 2603.02667v2…