PaperlayerView exact revision on arXiv

Source-linked AI summary

PixelDense: Dense Prediction as Representation Alignment for Pixel Diffusion

Lehan Yang, Daiqing Qi, Wenhao Zhang, Avery Li, Yiqing Yang, Yifan Li, Yu Kong, Haitian Zheng, Zhifei Zhang, Zhe Lin, Varun Jampani, Sheng Li

arXiv:2610.00483v1cs.CV

TL;DR

REPA has mostly aligned diffusion models with semantic encoders, despite evidence that spatial structure carries the alignment signal. PixelDense routes semantic and geometric teachers through separate projection streams, improving results across generation, reconstruction, and editing evaluations while leaving inference unchanged.

  • Problem

    REPA-style alignment has primarily used semantic encoders, while dense-prediction models that expose spatial structure remain underexplored as image-generation teachers.

    Source: §1 · p. 2

  • Method

    PixelDense routes DINOv2 and SAM2 through a semantic projection and Depth Anything v2 and Metric3D v2 through a geometric projection, with weight-space orthogonality between streams.

    Source: §1 · pp. 2–3

  • Results

    0.7927 to 0.8093 GenEval Overall on PixelGen, with PixelDense outperforming every single-teacher and unfactored multi-teacher variant and transferring to DeCo.

    Source: §1 · p. 2

  • Takeaways & Limitations

    PixelDense preserves source structure in reconstruction and SDEdit while keeping the original RGB generator and sampler at inference.

    Source: §1 · p. 2, §5 · p. 10

    Alignment is applied at one mid-network block; inversion-based and instruction-based editing and identity or 3D-geometry editing metrics remain future work.

    Source: E Broader Impact and Limitations · p. 21

Abstract

from arXiv · show

Representation alignment (REPA) accelerates diffusion transformer training, but its alignment targets are almost exclusively semantic encoders such as DINOv2 and CLIP. Recent analysis points to spatial structure, not global semantics, as the carrier of the alignment effect, yet dense-prediction foundation models trained to predict that structure remain overlooked as REPA targets. In pixel-space diffusion, SAM2, Depth Anything v2, and Metric3D v2 each outperform the DINOv2-only GenEval baseline, with the two geometric teachers leading the segmentation teacher. A flat sum of all four teachers, however, lands below the best single geometric teacher, as semantic and geometric gradients compete for one denoiser projection. We introduce PixelDense, which routes DINOv2 and SAM2 through a semantic projection stream, routes Depth Anything v2 and Metric3D v2 through a geometric projection stream, and adds a weight-space orthogonality penalty that keeps the two streams in disjoint subspaces. All four teachers are frozen during training and dropped at inference. Applied to PixelGen and DeCo with a single recipe, PixelDense improves GenEval, DPG-Bench, and HPS v2.1, raises PixelGen-XXL's GenEval Overall from 0.7927 to 0.8093, and beats every single-teacher and unfactored multi-teacher variant. In partial-noise reconstruction, independent panoptic, depth, and surface-normal probes show up to 53.1% PQ gain and 36.0% depth AbsRel reduction at $\tau=0.5$ across COCO and Flickr30K. From random initialization, PixelDense also reaches the baseline's peak GenEval 1.23x faster. In SDEdit editing on PIE-Bench, PixelDense keeps more of the source background and layout at every edit strength, raising background PSNR by up to 2.2 dB.

1 Introduction

REPA [61] has primarily used semantic encoders, while iREPA [46] points to spatial structure as the main alignment signal; dense-prediction teachers offer a way to bring that structure into image generation. PixelDense routes semantic and geometric teachers through separate projection streams to address their conflict under a shared projection.

1 Introduction

  • SAM2, Depth Anything v2, and Metric3D v2 each improve GenEval Overall over the DINOv2-only baseline, with both geometric teachers outperforming the segmentation teacher.The comparison uses a PixelGen model fine-tuned at 512×512.Source: §1 · p. 2
  • A flat four-teacher REPA sum through one projection falls below the best single geometric teacher because semantic and geometric teachers reward incompatible invariances.The paper frames teacher composition as a representation-routing problem.Source: §1 · pp. 2–3
  • PixelDense separates DINOv2 and SAM2 into a semantic stream and Depth Anything v2 and Metric3D v2 into a geometric stream, using weight-space orthogonality to prevent subspace collapse.The teachers are frozen during training and dropped at inference.Source: §1 · pp. 2–3
  • 0.7927 to 0.8093 GenEval Overall on PixelGen; PixelDense beats every single-teacher and unfactored multi-teacher variant and transfers to DeCo.The same alignment recipe is applied to both pixel-diffusion backbones.Source: §1 · p. 2
  • At τ = 0.5, panoptic PQ rises by 35.3% on COCO annotations and 53.1% on Flickr30K pseudo labels, while depth AbsRel falls by up to 36.0%.Standalone segmentation, depth, and surface-normal probes score better in every COCO and Flickr30K setting reported.Source: §1 · p. 2
  • PixelDense keeps more source background and layout at every SDEdit strength on PIE-Bench.The reported experiments also show improved compositional generation on PixelGen and DeCo and stronger structure preservation in partial-noise reconstruction.Source: §1 · pp. 2–3

2 Related Work

Prior pixel-space generators use RGB reconstruction targets, leaving image semantics and geometry implicit. Related work studies representation alignment and dense prediction in generation, while PixelDense uses dense-prediction models as training-time alignment teachers.

Diffusion models for image generation

  • Pixel-space methods remove the VAE bottleneck, but their supervision remains an RGB reconstruction target that leaves semantic and geometric structure implicit.The cited line includes Simple Diffusion [15], PixelFlow [5], JiT [26], DeCo [34], PixelGen [35], and other pixel-space methods.Source: §2 · p. 3

Representation alignment for diffusion models

  • REPA [61] aligns intermediate denoiser features with frozen encoders; iREPA [46] identifies spatial structure as the active ingredient, while later work extends the alignment recipe.Prior video work includes geometry-aware multi-encoder alignment, but sums teacher signals without addressing semantic–geometric conflict [23] [53].Source: §2 · p. 3

Dense prediction meets generation

  • Most dense-prediction-and-generation work repurposes diffusion features for perception or combines image and dense outputs; PixelDense instead uses frozen discriminative encoders as generator alignment teachers.The teachers are routed through separate semantic and geometric streams and dropped at inference.Source: §2 · p. 3
  • Figure 2 depicts the denoiser block-8 state feeding semantic and geometric projections, aligned respectively to DINOv2 and SAM2, and Depth Anything v2 and Metric3D v2.A weight-space orthogonality penalty keeps the projection streams from collapsing into a shared subspace.Source: Figure 2 · p. 4

3 Method

PixelDense extends REPA in pixel-space diffusion by separating semantic and geometric teacher alignment into parallel streams and regularizing their input projections to preserve distinct subspaces.

3.3 PixelDense: Factored Streams with Orthogonal Subspaces

  • PixelDense routes DINOv2 and SAM2 through a semantic stream and Depth Anything v2 and Metric3D v2 through a geometric stream, with separate projection bottlenecks.Both streams read the same denoiser token; four teacher heads map stream outputs to teacher feature widths.Source: §3 · p. 4, §3.3 · p. 5

3.1 Pixel Diffusion with Single-Teacher REPA

  • In pixel-space diffusion, each intermediate patch-grid token aligns to the matching spatial teacher feature through a cosine loss on projected denoiser features.The frozen teacher maps the clean image to a token-aligned feature grid and receives no gradient.Source: §3.1 · p. 4

3.2 Why a Single Alignment Stream Cannot Carry Dense Perception

  • A flat sum sends all four teacher losses through the same denoiser features, where semantic identity and viewpoint invariance conflict with geometric layout and surface-orientation signals.The method frames semantic–geometric factorization as a way to respect these distinct supervision signals.Source: §3.2 · pp. 4–5

3.3 PixelDense: Factored Streams with Orthogonal Subspaces

  • A weight-space penalty discourages the semantic and geometric input projections from reading the same denoiser directions, but does not constrain the full nonlinear streams to be globally orthogonal.The penalty uses projector weights and is data-free.Source: §3.3 · p. 5
  • The full objective combines flow-matching loss, factored alignment loss, orthogonality regularization, and auxiliary losses; teachers and alignment parameters are removed after training.The deployed model remains the original three-channel RGB generator with its original sampler.Source: §3.3 · p. 5

4 Experiments

Experiments use PixelGen-XXL as the main backbone, with DeCo testing transfer; both retain DINOv2 REPA during fine-tuning. Results cover generation, structure preservation, training from scratch, and editing.

4.2 Results

  • PixelDense improves PixelGen over its matched fine-tune on GenEval Overall, DPG-Bench, and HPS v2.1, raising GenEval Overall from 0.7927 to 0.8093.It also achieves the strongest GenEval Overall among the listed pixel-space backbones at this scale.Source: §4.2 · p. 6
  • Across datasets, reference types, probes, and noise levels, PixelDense shows stronger structural agreement than PixelGen; the gap narrows at τ = 0.9.The reconstruction comparison uses panoptic, depth, and surface-normal measures; Flickr30K panoptic scores are lower because OneFormer uses the COCO vocabulary.Source: Table 2 · p. 7, §4.2 · p. 7
  • Depth Anything v2 and Metric3D v2 score 0.8069 and 0.8060 on GenEval Overall, above SAM2 at 0.8020; full PixelDense reaches 0.8093.The flat four-teacher sum falls below the best single dense teacher, while each teacher contributes to the factored model.Source: §4.2 · pp. 7–8, Table 3 · p. 8

4.3 Ablation Studies

  • The ablations favor native-width projection streams, a mid-network alignment tap, slightly larger DINOv2 weighting, and the default λorth = 0.01.Both weaker and stronger orthogonality penalties lower GenEval Overall; the stronger penalty over-constrains the projections.Source: Table 4 · p. 8, §4.3 · p. 8

4.4 Training from Scratch

  • From random initialization, PixelDense reaches the baseline’s peak GenEval score roughly 1.23× faster and stays above it through the 100K-step horizon.The comparison uses matched optimizer, batch size, and data ordering at 128 × 128 resolution.Source: Figure 4 · p. 8, §4.4 · p. 8

4.5 Transfer to DeCo

  • Without retuning, PixelDense improves DeCo on both GenEval Overall and DPG-Bench over the matched DeCo fine-tune.The reported gain is smaller than on PixelGen, which the authors attribute as likely related to DeCo starting from a stronger model.Source: Table 5 · p. 9, §4.5 · p. 9

4.6 Transfer to Image Editing

  • On PIE-Bench SDEdit, PixelDense outperforms PixelGen on all six reported metrics at every noise level, though FID and DINOv2 gaps shrink at high τ.The evaluation compares background preservation, patch self-similarity, and FID under matched source images, prompts, noise, sampler, and step count.Source: Table 6 · p. 9, §4.6 · p. 9

4.7 Qualitative Results

  • Matched-prompt qualitative samples show PixelDense better preserves object counts, spatial relations, color binding, and scene geometry than PixelGen.The compared failure modes include object count, relative position, color binding, object boundaries, and perspective.Source: Figure 5 · p. 10, §4.7 · p. 10

5 Limitation and Conclusion

PixelDense factors semantic and geometric teachers into separate streams with weight-space orthogonality. Experiments report improved compositional alignment and structure preservation, while editing evaluation remains limited to SDEdit.

5 Limitation and Conclusion

  • PixelDense routes semantic and geometric teachers through separate streams and uses weight-space orthogonality to keep them from collapsing into one denoiser subspace.The framework improves compositional alignment on PixelGen and DeCo without inference-time cost.Source: §5 · p. 10
  • Reconstruction probes show stronger preservation of segmentation, depth, and surface-normal structure, while PIE-Bench SDEdit preserves more source background and layout at every edit strength.The conclusion describes PixelDense as promising for scenarios requiring strict structure preservation.Source: §5 · p. 10
  • The editing evaluation covers only SDEdit; inversion-based and instruction-based editing are left to future work.Source: §5 · p. 10

A Implementation Details

The implementation uses PixelGen-XXL and DeCo pixel-space backbones, frozen dense teachers, and separate semantic and geometric projection streams.

A.1 Pixel-Space Backbones

  • PixelGen-XXL and DeCo are 1.1B pixel diffusion transformers with the same patch grid, and both retain DINOv2 REPA during fine-tuning.The deployed sampler uses only RGB channels, so PixelDense adds no inference parameters.Source: §A.1 · p. 15

A.2 Frozen Dense Teachers

  • Four teachers operate on clean 512 × 512 images, remain frozen, and are dropped at inference; their features are resampled to the denoiser’s 16 × 16 patch lattice.The teacher groups are semantic and geometric, as summarized in Table 7.Source: Table 7 · p. 15, §A.2 · p. 15

A.3 Projection Streams and Teacher Heads

  • Both projection streams read block-8 denoiser tokens and use two-layer MLPs; their input maps are constrained by an orthogonality penalty.The semantic and geometric stream widths are 768 and 1024, respectively; teacher heads are single linear maps.Source: §A.3 · p. 15

A.4 Training Recipe

Fine-tuning uses a 10,000-step recipe with a combined flow-matching and alignment objective; DeCo uses the same recipe without retuning.

A.4 Training Recipe

  • Fine-tuning uses an effective batch size of 256, AdamW at learning rate 1e-5, a 200-step warmup, and 10,000 optimizer steps.The objective combines flow matching, four weighted cosine alignment losses, orthogonality regularization, and inherited LPIPS and Perceptual-DINO losses.Source: §A.4 · p. 15

A.5 Sampling

  • At inference, dense teachers and alignment parameters are dropped; sampling uses the Adam LM solver for 25 steps with classifier-free guidance scale 4.0.Resolution is 512 × 512 for PixelGen-XXL and DeCo and 128 × 128 for the from-scratch run.Source: §A.5 · p. 16

A.6 Datasets

The study uses the third-stage PixelGen training dataset for fine-tuning and evaluates generation, reconstruction, and dense-structure probes on specified datasets.

A.6 Datasets

  • Fine-tuning uses BLIP3-o-60K [4], PixelGen’s third-stage training dataset, without extra filtering.From-scratch training uses the same dataset at 128 × 128.Source: §A.6 · p. 16
  • Generation quality is evaluated with GenEval [9], DPG-Bench [17], and HPS v2.1 [54].Reconstruction evaluation uses COCO 2017 [28] validation and Flickr30K [39] test.Source: §A.6 · p. 16
  • Evaluation images are resized to 512 × 512 and center-cropped, with panoptic masks nearest-resampled to align pixel indices.Teachers, probes, and evaluation models are not retrained or fine-tuned.Source: §A.6 · p. 16

B.1 Text-to-Image Benchmarks

The appendix specifies the text-to-image benchmarks, partial-noise reconstruction probes, and spatial responses of the four dense teachers.

B.1 Text-to-Image Benchmarks

  • GenEval uses 553 prompts across six categories, while DPG-Bench and HPS v2.1 follow their released prompt and grading protocols.HPS v2.1 uses four 800-prompt style splits, with one image generated per prompt.Source: §B.1 · p. 16

B.2 Partial-Noise Reconstruction Probes

  • Partial-noise comparisons use identical Gaussian noise inputs for PixelGen and PixelDense at τ = 0.5, 0.7, and 0.9.COCO contributes 5,000 images and Flickr30K contributes 1,000 captioned images; segmentation, depth, and surface-normal probes assess reconstructions.Source: §B.2 · pp. 16–17

C.1 Semantic versus Geometric Teacher Features

  • DINOv2 responds to object regions and SAM2 to instance boundaries, while Depth Anything v2 and Metric3D v2 respond to depth-ordered silhouettes and surface-normal layout.Figure 6 shows per-patch RMS feature magnitudes, with yellow denoting high response.Source: Figure 6 · p. 17, §C.1 · p. 17

C.2 Training-Dynamics Diagnostic for Unfactored versus Factored Alignment

The diagnostics report that PixelDense maintains long-term co-improvement while reducing short-term coupling and balancing teacher-loss reductions. The section also checks probe robustness, style-split scores, additional examples, and asset use.

C.2 Training-Dynamics Diagnostic for Unfactored versus Factored Alignment

  • PixelDense retains high long-term cross-teacher loss correlation while reducing step-delta coupling and improving the balance of per-teacher loss reductions.The reported mean correlation is 0.9083 for PixelDense and 0.9057 for the naive sum; mean reduction rises from 72.13% to 73.40%.Source: §C.2 · pp. 17–19, Table 8 · p. 18, Table 10 · p. 19
  • The step-delta cosine falls from 0.7430 under the naive sum to 0.5966 under PixelDense, with the largest drop on the DINOv2–SAM2 pair.Depth Anything v2–Metric3D v2 remains strongly coupled, consistent with their shared 3D structure.Source: §C.2 · p. 18, Table 9 · p. 18

C.3 Per-Backbone Geometry Preservation Numbers

  • The reported panoptic backbones agree within 0.005 per cell, and median depth and surface-normal statistics preserve the mean-based rankings.The passage describes the agreement across probe results as evidence that the rankings are not driven by either backbone or tail outliers.Source: §C.3 · p. 19, Table 12 · p. 20, Table 13 · p. 20

C.4 HPS v2.1 Per-Style Breakdown

  • PixelDense matches or exceeds the matched PixelGen fine-tune on all four HPS v2.1 style splits and their average, with largest gains on Paintings and Concept-art.Each style split contains 800 prompts.Source: §C.4 · p. 20

C.5 Additional Qualitative Visualization

  • Additional 512 × 512 examples cover photographs, portraits, landscapes, and fantasy scenes; dense teachers are absent at inference.Additional partial-noise reconstruction comparisons are shown at the noise fractions used in Table 2.Source: §C.5 · p. 20

D Asset Licenses and Use

  • The paper uses public assets without modification and does not redistribute them.The training corpus and evaluation prompt sets use their official releases.Source: D Asset Licenses and Use · p. 20

E Broader Impact and Limitations

The broader-impact discussion notes potential misuse and inherited teacher biases, while the limitations identify a single alignment block and a narrow editing evaluation scope.

E Broader Impact and Limitations

  • PixelDense may be misused for fabricated imagery, identity-targeted content, or copyright-sensitive style mimicry, and inherits biases in perception-teacher training distributions alongside backbone biases.It does not change the backbone training dataset, and its teachers are dropped at inference.Source: E Broader Impact and Limitations · pp. 20–21
  • Alignment is applied at one mid-network denoiser block; multi-scale or multi-block alignment and scaling the backbone and corpus remain future extensions.The authors also identify broader teacher banks as a natural extension.Source: E Broader Impact and Limitations · p. 21
  • The editing evidence concerns SDEdit; inversion-based and instruction-based editing and post-edit identity or 3D-geometry metrics are left to future work.Under SDEdit, PixelDense keeps more of the source background and layout.Source: E Broader Impact and Limitations · p. 21

Related papers

Loading 2610.00483v1…