Source-linked AI summary
V-Co: A Closer Look at Visual Representation Alignment via Co-Denoising
Han Lin, Xichen Pan, Zun Wang, Yue Zhang, Chu Wang, Jaemin Cho, Mohit Bansal
TL;DR
Pixel-space diffusion lacks explicit semantic supervision, while existing visual co-denoising methods entangle design choices. V-Co isolates these choices in a unified JiT framework, identifies a four-part recipe, and reports stronger ImageNet-256 results than pixel-diffusion baselines and prior methods.
Problem
Pixel-level denoising is not explicitly designed to enforce high-level semantic structure, and existing co-denoising approaches leave essential design choices unclear.
Method
V-Co systematically studies pixel-space co-denoising by jointly denoising images and frozen DINOv2 patch features in a unified JiT-based testbed.
Results
V-Co outperforms the underlying JiT baseline and prior pixel-diffusion methods on ImageNet-256; 260M-parameter V-Co-B/16 matches 459M-parameter JiT-L/16 at FID 2.33 vs. 2.36.
Takeaways & Limitations
Effective visual co-denoising combines fully dual-stream processing, structural CFG masking, perceptual-drifting hybrid supervision, and RMS-based feature calibration.
Takeaways & Limitations
The study focuses on class-conditional ImageNet-256 generation and does not cover open-ended text-to-image synthesis or more structured multimodal tasks.
Abstract
from arXiv · showhide
Pixel-space diffusion has recently re-emerged as a strong alternative to latent diffusion, enabling high-quality generation without pretrained autoencoders. However, standard pixel-space diffusion models receive relatively weak semantic supervision and are not explicitly designed to capture high-level visual structure. Recent representation-alignment methods (e.g., REPA) suggest that pretrained visual features can substantially improve diffusion training, and visual co-denoising has emerged as a promising direction for incorporating such features into the generative process. However, existing co-denoising approaches often entangle multiple design choices, making it unclear which design choices are truly essential. Therefore, we present V-Co, a systematic study of visual co-denoising in a unified JiT-based framework. This controlled setting allows us to isolate the ingredients that make visual co-denoising effective. Our study reveals four key ingredients for effective visual co-denoising. First, preserving feature-specific computation while enabling flexible cross-stream interaction motivates a fully dual-stream architecture. Second, effective classifier-free guidance (CFG) requires a structurally defined unconditional prediction. Third, stronger semantic supervision is best provided by a perceptual-drifting hybrid loss. Fourth, stable co-denoising further requires proper cross-stream calibration, which we realize through RMS-based feature rescaling. Together, these findings yield a simple recipe for visual co-denoising. Experiments on ImageNet-256 show that, at comparable model sizes, V-Co outperforms the underlying pixel-space diffusion baseline and strong prior pixel-diffusion methods while using fewer training epochs, offering practical guidance for future representation-aligned generative models.
1. Introduction
V-Co systematically studies visual co-denoising in a unified pixel-space JiT framework to identify essential design choices for representation alignment. Its four-part recipe improves ImageNet-256 generation relative to pixel-diffusion baselines and prior methods.
- Pixel-space denoising does not explicitly enforce high-level semantic structure, limiting the sample efficiency of semantic representation learning.
- V-Co jointly denoises image pixels and patch-level semantic features from a frozen pretrained visual encoder within a controlled JiT-based testbed.
- A fully dual-stream JiT best preserves feature-specific computation while enabling flexible cross-stream interaction.
- Structural semantic-to-pixel masking defines unconditional CFG prediction, while a perceptual-drifting hybrid loss combines instance-level alignment with distribution-level regularization.
- RMS-based feature rescaling calibrates the semantic and pixel streams through an equivalent semantic-stream noise-schedule shift based on SNR matching.
- 260M-parameter V-Co-B/16 matches 459M-parameter JiT-L/16 at FID 2.33 vs. 2.36, while V-Co-L/16 and V-Co-H/16 outperform 2B-parameter JiT-G/16 at FID 1.71 vs. 1.82.
2. Related Work
Prior work spans pixel-space diffusion, representation alignment, and joint denoising of images with semantic or structured signals. V-Co builds on these directions while using pixel space as a controlled setting for studying co-denoising and representation alignment.
- Pixel-space diffusion generation: JiT and related methods show that scalable Transformer denoisers can make pixel-space diffusion competitive with latent diffusion while avoiding autoencoder bottlenecks.
- Pixel-space diffusion generation: V-Co adopts pixel-space diffusion to avoid autoencoder-induced bottlenecks and latent-space biases when studying representation alignment.
- Representation alignment for diffusion training: Representation-alignment methods such as REPA inject pretrained visual knowledge into diffusion training to improve convergence and sample quality.
- Visual co-denoising and joint generation: Joint denoising methods model image latents together with semantic features or other structured signals across image, video, audio-visual, and robotics settings.
3. A Closer Look at Visual Co-Denoising
The paper establishes a unified JiT-based pixel-space testbed for visual co-denoising and isolates four design choices governing its effectiveness: architecture, CFG conditioning, auxiliary losses, and feature calibration.
- Study Design: Controlled ablations compare shared-backbone and dual-stream architectures, unconditional CFG designs, auxiliary objectives, and feature rescaling.The study uses a unified JiT-based testbed to isolate each component’s contribution.
- 3.1. Formalizing Visual Co-Denoising: Visual co-denoising jointly denoises image pixels and patch-level semantic features from a frozen pretrained visual encoder.The framework extends pixel-space diffusion with a semantic feature stream under a shared diffusion process.
- 3.2. Architecture: Preserving feature-specific processing while enabling cross-stream interaction favors token concatenation and a fully dual-stream architecture.The fully dual-stream variant achieves the best FID of 8.86 among the compared architectural designs.
- 3.3. Unconditional Prediction for CFG: Structural semantic-to-pixel masking with joint conditioning dropout provides the most effective unconditional prediction for CFG.Blocking semantic information from reaching the pixel branch is more effective than input-level dropout.
- 3.4. Auxiliary Losses: A perceptual-drifting hybrid loss best combines instance-level semantic alignment with distribution-level regularization.The hybrid objective achieves guided FID 2.44, compared with 2.73 for perceptual loss and 2.85 for drifting loss.
- 3.5. Feature Calibration: RMS-based feature rescaling calibrates semantic and pixel streams with different representation scales and substantially improves guided generation.Removing RMS scaling worsens guided FID from 2.52 to 5.28, while noise-schedule shifting gives 2.93.
4. Full Recipe and SoTA Comparison
The full V-Co recipe combines dual-stream processing, structural CFG masking, hybrid semantic supervision, and RMS-based calibration. On ImageNet 256×256, it matches or surpasses larger pixel-space diffusion baselines and other strong methods.
- V-Co combines a fully dual-stream JiT backbone, structural semantic-to-pixel masking with joint dropout, a perceptual-drifting hybrid loss, and RMS-based feature rescaling.These components address stream interaction, guidance formation, semantic supervision, and cross-stream calibration, respectively.
- V-Co-B/16 with 260M parameters matches JiT-L/16 with 459M parameters at FID 2.33 vs. 2.36.
- V-Co-L/16 and V-Co-H/16, trained for 500 and 300 epochs, outperform JiT-G/16 with 2B parameters at FID 1.71 vs. 1.82.
- RMS scaling matches or outperforms Latent Forcing while avoiding its separate pixel and DINOv2 noise schedules.The comparison supports a simple calibration strategy for representation-aligned pixel-space generation.
5. Conclusion
V-Co identifies four ingredients for effective visual co-denoising and combines them into a simple, scalable recipe for representation-aligned pixel-space generation.
- V-Co uses a fully dual-stream backbone to preserve feature-specific computation while enabling cross-stream interaction.
- Structural semantic-to-pixel masking defines unconditional classifier-free guidance prediction.
- A perceptual-drifting hybrid loss combines instance-level alignment with distribution-level regularization.
- RMS-based feature rescaling supports stable co-denoising, while the combined recipe improves semantic alignment and generative quality on ImageNet.
A. Experiment Setup Details
The experiments progressively extend a minimal JiT-based co-denoising setup with visual features, conditioning choices, architectural comparisons, and configuration settings.
- The configuration experiments cover architecture, feature preprocessing, training, and sampling settings for V-Co-B/L/H.
- The minimal setup adds a DINOv2 branch that preprocesses DINOv2-Base features using dataset-level statistics from RAE.
- Conditioning uses independent dropout initially, with class-label dropout 0.1 and DINOv2 feature dropout 0.2.
- Architectural comparisons include baseline, single-stream, and dual-stream variants while fixing the pixel-stream depth at 12 JiT blocks.
- From Sec. 3.3 onward, the setup adopts joint class-label and DINOv2-feature dropout with probability 0.1 and semantic-to-pixel attention masking.
B. Additional Ablations
Additional ablations examine architectural allocation, dropout, similarity-based gating, hybrid-loss hyperparameters, and DINOv2 model-size settings.
- A balanced allocation of 6 feature-specific and 6 shared blocks generally performs best among 12-block single-stream designs.
- Jointly dropping class labels and DINOv2 features generally outperforms independently dropping either conditioning source.The stated rationale is closer alignment between training and inference when unconditional prediction is structurally defined.
- The default similarity-based gating achieves the best guided FID compared with a simplified scalar gate.
- Hybrid-loss studies sweep repulsion temperature, gate temperature, and hybrid-loss coefficient while reporting unguided and guided FID and IS.
- DINOv2 model-size experiments sweep the diffusion loss coefficient λ_d over {1e−3, 1e−2, 1e−1, 1}.
C. Generated Samples
The appendix presents uncurated 256×256 ImageNet samples from V-Co-H/16 after 300 training epochs using the CFG value associated with the reported FID.
- The samples are uncurated ImageNet 256×256 generations from V-Co-H/16 after 300 epochs of training.
- Samples are conditioned on specified classes.
- The visualization uses CFG 1.5, the same value used to obtain the reported FID of 1.71.
D. Limitation and Future Work
V-Co’s evaluation and design remain bounded by class-conditional ImageNet-256 experiments, reliance on DINOv2 features, and intentionally limited auxiliary supervision.
- Evaluation is limited to class-conditional ImageNet-256 generation, leaving open-ended text-to-image synthesis and structured multimodal tasks untested.The authors identify extending V-Co beyond ImageNet-style class conditioning as an important direction.
- V-Co relies on DINOv2 features, so co-denoising dynamics may depend on the teacher representation’s quality, inductive biases, and spatial granularity.The paper proposes exploring alternative semantic feature sources.
- The minimalist method omits stronger auxiliary supervision, including combinations of REPA-style objectives with the perceptual-drifting hybrid loss.Future work could study how the V-Co recipe interacts with richer objectives and stronger supervision.