PaperlayerView exact revision on arXiv

Source-linked AI summary

Iris-3B: Going Beyond the Latent with Pixel-Space Diffusion Training, Conversion and Fine-Tuning

Hanqiu Li Cai, Chema Garabito

arXiv:2610.09450v1cs.CVcs.AI

TL;DR

Pixel-space diffusion avoids the lossy VAE used by latent models, motivating the question of whether it transfers better to detail-critical downstream tasks. The paper trains Iris-3B from scratch and converts FLUX.2 Klein to pixel space, then evaluates both families on depth and restoration. It finds no significant downstream improvement from a pixel-space prior, while Iris-3B scales to 3B parameters for text-to-image generation.

  • Problem

    Whether pixel-space models transfer better to detail-critical downstream tasks remains untested, despite the structural VAE bottleneck in latent backbones.

    Source: §01 · p. 2

  • Method

    The paper trains Iris-3B from scratch and converts FLUX.2 Klein base 4B to pixel space, then fine-tunes both families for monocular depth and image restoration.

    Source: §01 · p. 2

  • Results

    No significant improvement from a pixel-space prior is found: Iris-3B is level with latent FLUX.2 Klein for depth, while the converted model falls behind it and trails slightly on restoration.

    Source: §01 · p. 2, §06 · p. 10

  • Takeaways & Limitations

    Iris-3B demonstrates pixel-space pretraining at 3B parameters, but these downstream comparisons show no significant advantage over latent FLUX.2 Klein.

    Source: §01 · p. 2, §06 · p. 10

    The downstream comparisons are confounded; Iris-3B has no latent twin, so its depth comparison also varies in model size, pretraining data, and compute.

    Source: §06 · p. 10

Abstract

from arXiv · show

Pixel-space diffusion models avoid the lossy VAE of latent models, which suggests an advantage on downstream tasks where fine-grained detail matters. We test this claim along both routes to a pixel-space backbone. We pretrain Iris-3B, a 3B-parameter pixel-space text-to-image transformer, from scratch through a $256\to512\to1024$ curriculum, after first ablating the prediction target and representation alignment at $256^2$ to decide what to scale. We also convert a pretrained latent model, FLUX.2 Klein base 4B, to pixel space. We fine-tune both families for monocular depth estimation and for image restoration/super-resolution. We find no significant improvement from using a pixel-space generative prior. Fine-tuned for depth with one matched direct-regression recipe, Iris-3B is level with the latent FLUX.2 Klein and the converted pixel FLUX.2 Klein falls behind it, and on $4\times$ DIV2K restoration neither pixel model beats a latent FLUX.2 Klein fine-tune, the converted one trailing it slightly. We document the recipes, the failure modes and the remaining confounds behind this negative result. Nevertheless, Iris-3B shows that pixel-space pretraining with the pixel-transformer (PiT) head of PixelDiT scales to 3B parameters and to text-to-image quality competitive with latent models, matching Qwen-Image on OneIG under the official evaluators at $1024^2$. We release its weights and training code in the hope that they help pave the way for further work on pixel-space generation.

01 · Introduction

The paper tests whether pixel-space generative backbones transfer better to detail-critical tasks, using both scratch pretraining and latent-to-pixel conversion. Neither route yields a significant downstream improvement, while Iris-3B demonstrates pixel-space text-to-image pretraining at 3B parameters.

01 · Introduction

  • Pixel-space models had been judged by generation or understanding, leaving their value as backbones for other vision tasks untested.The motivation is that dense outputs from latent backbones must pass through a VAE optimized for natural-image reconstruction.Source: §01 · p. 2
  • The study tests pixel-space backbones from two routes: scratch pretraining Iris-3B and converting pretrained latent FLUX.2 Klein base 4B.Both model families are fine-tuned for monocular depth estimation and image restoration with super-resolution, then compared with latent counterparts.Source: §01 · p. 2
  • For depth, Iris-3B is level with latent FLUX.2 Klein, while converted FLUX.2 Klein falls behind; on restoration, the converted model trails its latent counterpart slightly.The comparisons retain confounds, which the paper documents rather than claiming to resolve.Source: §01 · p. 2
  • A controlled 256px ablation campaign compares prediction targets and representation-alignment strategies to select a pixel-space pretraining recipe.The targets are x versus v; alignment variants include REPA, iREPA, DINOv3 targets, and Self-Flow, each compared with its own control at matched steps.Source: §01 · p. 2
  • For FLUX.2 Klein conversion, the input interface is identified as an obstacle, addressed with gain-matched ridge initialization and a scaled trunk learning rate.The proposed changes are described as removing that obstacle.Source: §01 · p. 2
  • Iris-3B is an open 3B-parameter pixel-space text-to-image model trained from scratch, with weights and training code released.Its training follows a 256 →512 →1024 curriculum and reaches GenEval 0.798 and DPG 86.52 under the official evaluators at 1024².Source: §01 · p. 2

02 · Related Work

Related work covers pixel-space generation, conversion of latent models, generative priors for dense prediction and restoration, and representation alignment. These lines of work provide the approaches and precedents against which the paper positions its experiments.

Pixel-space generation

  • Pixel-space generation has progressed from cascades and pixel-level U-Nets to transformer approaches for class-conditional and text-to-image generation.Methods include large-patch clean-image prediction, patch-level and pixel-level transformers, pixel decoders, neural fields, perceptual losses, and low-rank noise prediction.Source: §02 · pp. 2–3

Latent-to-pixel conversion

  • Latent-to-pixel conversion replaces the VAE interface in pretrained models, with prior work reporting conversions that approach or exceed their latent parents.L2P, Jiang et al., and AsymFlow use different conversion strategies, including shallow-layer training, systematic recipe ablations, and low-rank alignment.Source: §02 · p. 3

Generative priors for dense prediction

  • Generative priors have been adapted to dense prediction, including latent text-to-image models fine-tuned for depth and pixel-space DiTs trained for point maps.Related work also explores single-step depth fine-tuning, feature alignment, and raw point-map patch denoising.Source: §02 · p. 3

Generative restoration

  • Diffusion-based restoration conditions a pretrained text-to-image generator on a low-quality image, with approaches ranging from multi-stage systems to one-step methods.Examples include restoration modules followed by detail regeneration, semantic guidance, and score-prior initialization with adversarial training.Source: §02 · p. 3

Representation alignment

  • Representation-alignment methods train DiT features against external encoder targets or self-supervised objectives, and the paper ablates REPA, iREPA, and Self-Flow.REPA aligns intermediate features with frozen DINOv2 features; iREPA emphasizes target spatial structure, while Self-Flow uses an objective built into flow matching.Source: §02 · p. 3

03 · Iris-3B: Pretraining a Pixel-Space Model

The section selects v-prediction and DINOv2 REPA from controlled 256px ablations, then scales that recipe into Iris-3B. The resulting model is competitive with strong latent models on official 1024² benchmarks, with strengths and gaps varying by benchmark.

3.1 Ablations at 256px

  • At 256px, v-prediction is stable and tied with x-prediction on GenEval, while x-prediction is worse on DPG and FID; the comparison also changes the timestep distribution.The x-prediction arm uses JiT’s logit-normal (0.8, 0.8) distribution rather than the default (0.0, 1.0), so the contrast does not isolate the target.Source: §3.1 · pp. 3–4
  • DINOv2 REPA is strongest overall; DINOv3 lowers GenEval and worsens FID, iREPA’s temporary FID gain nearly vanishes by 250K, and Self-Flow remains far below control.DINOv2 REPA leads on GenEval at every milestone and is level with the best arm on DPG and CLIP; the small final DPG lead for DINOv3 is within evaluation noise.Source: §3.1 · p. 4, Figure 3 · p. 4

3.2 Scaling to Iris-3B

  • Iris-3B scales the selected recipe to 3B parameters with a hybrid dual/single-stream trunk over 16×16 pixel patches and a PiT pixel head.The trunk has 8 dual-stream blocks and 16 single-stream blocks; the four-block PiT head decodes trunk output to pixels.Source: §3.2 · pp. 4–5, Figure 4 · p. 5

3.3 Training

  • The model follows a 256→512→1024 curriculum with hybrid Muon/AdamW optimization, then receives 1024-resolution fine-tuning on a curated image set.At 1024, the learning rate is halved and global batch reduced to 512; the final pretraining checkpoint is fine-tuned at a constant learning rate of 4×10^-5.Source: §3.2 · p. 4, §3.3 · pp. 5–6
  • Training uses a rectified-flow velocity objective with DINOv2 REPA at 256px, plus resolution-specific data buckets and noise shifts.The pretraining collection mixes real photographs, synthetic images and rendered-text images; REPA weight is 0.5 at 256px and zero afterward.Source: §3.3 · p. 5

3.4 Evaluation

  • Iris-3B is competitive with strong latent models on official benchmarks, matching i1 on DPG and Qwen-Image on OneIG-EN while trailing several models on GenEval and long-text rendering.It is strongest on single objects, two objects and colors, and weakest on position and counting; i1 and Z-Image use prompt rewriting.Source: §3.4 · p. 6, Table 2 · p. 6

04 · Latent-to-Pixel Conversion

The paper converts FLUX.2 Klein base 4B from latent to pixel space, replacing its VAE with pixel patches and a new input projection and pixel head. It develops an initialization and learning-rate recipe for the conversion, then uses the PiT-head model for downstream evaluations.

4.1 Setup

  • The conversion replaces the VAE with 16×16 pixel patches, adds an input projection and pixel head, and trains the full model to predict clean images.The training data mix real photographs and step-distilled FLUX.2 Klein 4B images in a 1:1 ratio; the PiT head is used for downstream evaluations to match Iris-3B.Source: §4.1 · p. 6

4.2 Ridge initialization and trunk LR

  • In small-batch probes, a random input projection feeds the trunk unfamiliar inputs; the 1K-step result is patch-grid noise, and the unmodified recipe needs over ten times as many steps to approach the improved recipe’s loss.The trunk was trained on VAE-latent embeddings, and the best per-patch linear pixel-to-embedding map explains only 14% of their variance.Source: §4.2 · p. 6
  • The gain-matched ridge map projects each RGB patch toward its matching latent-token embedding, while the trunk learning rate is scaled to equalize relative updates.For FLUX, the scaled trunk rate is 0.12× the head rate; gain matching avoids the pixel-noise amplification of the held-out-R2-tuned fit.Source: §4.2 · p. 6
  • Together, gain-matched ridge initialization and a scaled trunk learning rate turn 1K-step patch-noise probes into photorealistic, prompt-aligned images.Either change alone more than halves 1K-step loss but does not produce coherent images; the probes use global batch 8.Source: §4.2 · pp. 6–7

4.3 Production conversions

  • The production conversion uses a PiT-head model after 30K steps for downstream tasks, but its GenEval, DPG, and FID scores were not measured.Both converted heads produce photorealistic images and render short words correctly.Source: §4.3 · p. 7

05 · Downstream Tasks

The section tests whether pixel-space generative priors improve dense prediction by fine-tuning pixel-space and latent backbones for depth and restoration. The tested pixel-space models show no clear advantage over latent FLUX.2 Klein.

5.1 Protocol

  • The comparison tests depth and restoration because both require dense, pixel-aligned outputs, where VAE compression may discard fine detail.Iris-3B has no latent twin, so its depth comparison with FLUX.2 Klein also mixes representation space with differences in model size, pretraining data, and compute.Source: §5.1 · p. 7

5.2 Monocular depth

  • Depth is predicted in one deterministic forward pass, using a masked L1 loss plus a weighted L1 loss on depth gradients over valid pixels.All three parents use the same 10K-step direct-regression recipe; evaluation covers five zero-shot benchmarks and Hypersim soft edge error.Source: §5.2 · pp. 7–8
  • Iris-3B is level with latent FLUX.2 Klein on mean depth metrics, while converted Klein pixel trails latent Klein on every benchmark except KITTI, where they are level.Iris-3B is better on KITTI and DIODE and has the lowest edge error, but is worse on NYUv2, ETH3D, and ScanNet; its benchmark differences cancel in the mean.Source: §5.2 · p. 8, Table 4 · p. 9
  • Both pixel-space models show patch-grid artifacts in raw depth predictions, strongest for Iris-3B, while the latent model has no grid.The patchified design can let neighboring patches disagree at borders; the artifacts are small relative to depth error but work against fine spatial detail.Source: §5.2 · p. 9, Table 5 · p. 9

5.3 Restoration and super-resolution

  • In matched DIV2K 4× restoration, converted Klein pixel is worse on most metrics, most clearly SSIM and NIQE, and level on LPIPS against latent Klein.Each arm is a single run without a significance test, so the small differences indicate no advantage rather than a reliable loss.Source: §5.3 · p. 9, Table 6 · p. 10
  • On the same restoration pairs, Iris-3B is slightly below both FLUX.2 Klein models on fidelity and DeQA, between them on NIQE, and best on MUSIQ.This comparison is unmatched because its parent checkpoint, batch, schedule, and budget differ, and inference adds wavelet colour correction and tiling; overall, the adapted pixel models are competitive with open-source restoration methods.Source: §5.3 · pp. 9–10

06 · Conclusion

Iris-3B scales pixel-space pretraining to 3B parameters, but the downstream comparisons show no significant improvement from a pixel-space prior. The authors emphasize remaining confounds and propose directions for future work.

06 · Conclusion

  • Neither pixel model improves on latent FLUX.2 Klein: Iris-3B is level for depth, while the converted model trails it for depth and slightly for restoration.The authors conclude that they find no significant improvement from a pixel-space generative prior.Source: §06 · p. 10
  • Iris-3B reaches 3B parameters without a VAE and is competitive with strong latent models for text-to-image generation.The authors release its weights and training code for further pixel-space generation work.Source: §06 · p. 10
  • Downstream comparisons are confounded by mismatched model size, pretraining data, compute, conversion duration, and single short-budget runs without significance tests.Iris-3B restoration is also unmatched to the FLUX.2 Klein arms, and dense predictions show faint patch artifacts.Source: §06 · pp. 10–11
  • Future work includes improving Iris-3B, removing the patch grid with overlapping receptive fields, and testing pixel-space editing against a matched latent model.The proposed matched editing comparison would address this study’s remaining confounds.Source: §06 · p. 11

A · Text-to-Image Samples

The samples show Iris-3B outputs generated directly in pixel space at about one megapixel across native aspect ratios. Their prompts span portraits, art styles, natural scenes, landmarks, and imagined subjects.

A · Text-to-Image Samples

  • The final EMA checkpoint after supervised fine-tuning is shown at native aspect ratios, using CFG 3 and 100 sampling steps.Each image uses one fixed seed and comes from a single rendering of a broad prompt set.Source: A · Text-to-Image Samples · p. 13
  • Prompts include detailed portraits, from a bearded fisherman to a Maasai elder and a young woman with glitter freckles.Other portrait prompts specify a mosaic peacock, a scientific mushroom illustration, or a Frida Kahlo-inspired scene.Source: A · Text-to-Image Samples · pp. 14–15
  • Other prompts range from Petra and Chiang Mai lanterns to aurora-lit Lofoten, a floating airship city, and a cloud-swimming whale.The gallery also includes a volcanic eruption and a vintage coastal drive.Source: A · Text-to-Image Samples · pp. 13, 15

B · Additional Depth Results

The additional depth figure presents predictions from three model arms across five zero-shot benchmarks. Its caption specifies the prediction scaling and alignment used for display.

B · Additional Depth Results

  • The figure shows predictions from three model arms on NYUv2, KITTI, ETH3D, ScanNet, and DIODE.Two evenly spaced images from each dataset are shown, top to bottom, and were not selected by quality.Source: B · Additional Depth Results · p. 16, Figure 9 · p. 16
  • Panels show raw relative-log-depth predictions as inverse depth, with near regions bright and each prediction scaled to its own 2nd–98th percentile.No ground-truth alignment is applied.Source: B · Additional Depth Results · p. 16
  • The figure caption identifies the predictions as results after 10K steps of the same direct-regression recipe.Source: Figure 9 · p. 16

C · Additional Restoration Results

The additional restoration figures show further DIV2K validation examples and real-world low-quality inputs. The captions identify the degradation setting and note that the real-world examples have no ground truth.

C · Additional Restoration Results

  • The images and regions in these panels were chosen after viewing the outputs.Settings match Fig. 8, except Iris-3B panels in Fig. 12 use an earlier 5K-step checkpoint.Source: C · Additional Restoration Results · p. 17
  • Further DIV2K validation examples are shown with native Real-ESRGAN degradations and with the Tab. 6 pairs.These examples continue across the two restoration figures.Source: Figure 10 · p. 17, Figure 11 · p. 17
  • Real-world examples include low-quality images from RealLQ250 and four inputs from the public HYPIR repository, both without ground truth.Source: Figure 12 · p. 18, Figure 13 · p. 19
Loading 2610.09450v1…