Source-linked AI summary
From RGB Generation to Dense Field Readout: Pixel-Space Dense Prediction with Text-to-Image Models
Zanyi Wang, Xin Lin, Haodong Li, Dengyang Jiang, Yijiang Li
TL;DR
Dense prediction needs pixel-correct, task-native fields rather than RGB-like targets rendered through a generative decoder. ReChannel preserves the DiT’s RGB input pathway but directly reads task fields from adapted patch tokens, achieving state-of-the-art results on several benchmarks and up to 2.48× faster inference than edit-plus-latent decoding. The approach depends on a strong pretrained prior and remains evaluated within a limited scope.
Problem
Existing generative dense predictors reuse RGB-generation priors through target-side VAE representations and decoding, although dense prediction requires task-native pixel-space fields rather than rendered RGB content.
Method
ReChannel retains the VAE encoder, adapts the DiT token field with task LoRA, and uses a shared token-local linear head to read out task-native target patches without a target-side decoder.
Results
Across six tasks and more than a dozen benchmarks, ReChannel reaches state-of-the-art results on trimap-free matting, KITTI depth, and referring segmentation while running up to 2.48× faster than edit-plus-latent decoding.
Takeaways & Limitations
Generative pretraining can support dense perception by organizing an RGB-native field for direct task-native readout rather than by rendering the target.
Takeaways & Limitations
The method relies on a strong pretrained prior, and extension beyond FLUX-Klein and pixel-aligned targets remains future work.
Abstract
from arXiv · showhide
Large-scale text-to-image models are attractive backbones for dense prediction because RGB generation pretraining learns rich semantic, structural, and geometric priors. Existing generative and editing approaches reuse these priors by casting dense prediction as target generation: annotations such as depth, normals, alpha mattes, masks, and heatmaps are encoded into an RGB-trained VAE latent space and decoded back as image-like targets. We argue this inherits more of the generative output interface than dense prediction requires: unlike RGB synthesis, dense prediction asks for pixel-correct, task-native fields on the same image plane, not new RGB content to be rendered. Our key observation is that a pretrained DiT already organizes RGB inputs through a patch-to-token-to-patch lattice on the image plane, so each token indexes a fixed output patch whose channels can carry task-native quantities instead of RGB appearance. We instantiate this as ReChannel: we keep the VAE encoder for the DiT's input distribution but drop the target-side decoder, adapt the frozen DiT with task LoRA, and map each token to its p x p x K_t pixel-space patch through a shared token-local linear head--about 33K parameters, no spatial mixing. Using FLUX-Klein, we evaluate on six dense prediction tasks and over a dozen benchmarks. This minimal interface sets new state-of-the-art on trimap-free matting, KITTI depth, and referring segmentation, and stays competitive on normals, saliency, and pose. In a matched 4B setting it is more accurate and 2.48x faster than an edit-plus-latent-decode counterpart--dense perception can benefit from generative pretraining without inheriting its output interface.
1. Introduction
ReChannel argues that dense prediction should read task-native pixel fields directly from a pretrained DiT’s patch-aligned token field rather than generate targets through a rendering interface. A token-local readout with lightweight adaptation achieves strong results across diverse tasks while avoiding target-side decoding.
- Dense prediction estimates pixel-aligned geometry, masks, mattes, and heatmaps from RGB observations.
- RGB generation reconstructs appearance, whereas dense prediction re-expresses evidence as task-native fields such as geometry, opacity, or heatmaps.Task-adapted fields occupy more compact subspaces than the high-dimensional RGB input field.
- ReChannel reinterprets each DiT output patch’s channels as task-native quantities instead of RGB appearance.The patch-to-token-to-patch structure supplies spatial carriers aligned with the image plane.
- The method retains the VAE encoder, adapts the DiT with lightweight task LoRA, and maps each token to its corresponding target patch through a shared token-local linear projection.Dense targets bypass the VAE, eliminating a target-side reconstruction path and high-capacity spatial decoder.
- Across six dense prediction tasks and more than a dozen benchmarks, one token-local readout reaches state-of-the-art results across geometry, matting, language-conditioned segmentation, and pose heatmaps.Controlled ablations attribute the gains to the interface rather than head capacity or scale.
2. Related Work
Related work moves from task-specific decoders and foundation visual features toward reusing text-to-image generative priors for dense perception. Prior approaches retain generative output interfaces, while patch-indexed and pixel-space generation motivate task-native readout interfaces.
- Traditional dense prediction pairs task-specific target representations and output heads with increasingly transferable visual backbones.
- Text-to-image models have been adapted to dense perception tasks including monocular depth, surface normals, and general dense prediction.These approaches reuse geometric and perceptual structure learned during RGB synthesis pretraining.
- A generalist line expresses diverse vision tasks as RGB outputs, preserving a generative interface for dense prediction.
- ViT and DiT patch-indexed token grids, together with pixel-space generators, motivate modeling image-plane quantities without relying on a VAE decoder.The cited pixel-space interface had previously been demonstrated for generation rather than dense prediction.
3. Method
ReChannel reframes dense prediction as direct field readout from a pretrained T2I token grid rather than target generation through a VAE output path. It preserves the RGB input interface, adapts the backbone with task LoRA, and maps spatial tokens directly to task-native pixel patches.
- Design rationale: The method treats the DiT patch lattice as a spatial carrier whose channels can change from RGB appearance to task-native quantities.This output interface is used across geometry, masks, language-conditioned segmentation, and heatmaps.
- Interface: ReChannel preserves the pretrained T2I model’s RGB input pathway while removing the target-side generative reconstruction path.Dense targets remain outside the VAE and are supervised directly as pixel-space fields.
- Backbone adaptation: Task-specific LoRA adapts the frozen DiT token field toward the semantics of each dense prediction task.The backbone runs deterministically, while the pretrained transformer weights remain frozen.
- Pixel readout: A shared token-local linear projection maps each spatial token to its corresponding p × p × K_t target patch.Tiling the patches produces the final prediction, with p denoting patch size and K_t the task output-channel count.
- Pixel readout: The readout uses no inter-token spatial mixing, so spatial structure comes from the adapted token field rather than a learned spatial decoder.The same token-local form supports task-native channels such as scalar fields, three-channel normals, and pose heatmaps.
4. Experiments
ReChannel is evaluated across diverse dense prediction benchmarks using a frozen FLUX-Klein backbone with task LoRA and a token-local readout. It achieves strong results across geometry, matting, referring segmentation, pose, and saliency while avoiding target-side decoding overhead.
- Geometry: ReChannel-9B achieves the best KITTI absRel of 0.063, improving over Edit2Perc by 0.016.
- Trimap-free matting: ReChannel-9B sets new state-of-the-art matting results on P3M-500-P and P3M-500-NP with SAD scores of 5.69 and 6.67.These improve over ViTAE-S by 0.55 and 0.92 SAD, respectively.
- Referring segmentation: ReChannel-9B reaches 82.0 average cIoU and is best on all eight RefCOCO splits for referring segmentation.ReChannel-4B already exceeds FCLM-7B, achieving 80.3 versus 79.4 average cIoU.
- Pose and saliency: ReChannel-9B reaches 79.2 AP on COCO pose, surpassing ViTPose-L by 0.9 AP, and leads saliency on both Fmax and MAE.Saliency scores are Fmax 0.944 and 0.968, with MAE 0.018 and 0.017 on DUTS-TE and ECSSD.
- Diagnostic ablations: With only a trained linear head, normals collapse to 44.30°/43.55°/48.16° and matting reaches SAD 180.97/170.92, showing LoRA adaptation is necessary.
- Diagnostic ablations: Random initialization remains substantially worse, including 22.90° on NYU normals and matting SAD 11.56/12.32.The authors attribute this gap to reliance on a strong pretrained prior rather than token-grid structure alone.
- Efficiency: A token-local readout matches or exceeds larger heads and full fine-tuning, while latent, VAE-decoded, and edit outputs are consistently less accurate.The matched 4B backbone runs in 47.7 ms per image, compared with 74.4 ms for latent/VAE-frozen variants and 118.1 ms for editing.
5. Conclusion
ReChannel revisits the interface between text-to-image priors and dense prediction by removing target-side VAE rendering and directly reading task-native pixel patches from adapted DiT tokens. Across six tasks and more than a dozen benchmarks, this interface reaches state-of-the-art results on several tasks and runs up to 2.48× faster than an edit-plus-latent-decode counterpart.
- ReChannel removes the target-side VAE and reads task-native pixel patches from the adapted DiT token field using a token-local linear head.
- Across six tasks and more than a dozen benchmarks, ReChannel reaches state-of-the-art results on trimap-free matting, KITTI depth, and referring segmentation.
- 2.48× faster, ReChannel outpaces an edit-plus-latent-decode counterpart while maintaining the reported accuracy advantage.
- The results suggest dense perception benefits from the RGB-native field organized by generative pretraining rather than its target-side rendering interface.
- Extending ReChannel beyond FLUX-Klein and pixel-aligned targets remains future work.