Source-linked AI summary
DAGS: Disentangled Appearance-and-Geometry Steering of a Frozen Image DiT for Temporally Stabilized Generative Rendering
Karthik Mohan Kumar, Damian Andrysiak, Pedro Antonio Pena, Kunal Tyagi, Rama Harihara
TL;DR
Precisely controlling diffusion transformers is difficult, and the condition used can affect faithfulness to a target. DAGS steers a frozen image DiT with disentangled geometry and appearance conditioning plus temporal stabilization; it outperforms the cited baselines on reconstruction and perceptual stability in the reported matched-input comparison.
Problem
Diffusion transformers generate high-fidelity images, but precise control remains difficult and faithfulness depends heavily on how conditions are supplied.
Source: §1 · p. 2Method
DAGS steers a frozen image DiT using separate geometry and appearance encoders, with recurrent lighting stabilization and optional training-free temporal guidance.
Source: §1 · p. 2, §2 · p. 2, §2.3 · p. 2Results
DAGS wins reconstruction and temporal-LPIPS stability on all nine scenes, with +8.6 dB PSNR over OIDN and +10.1 dB over RGB↔X, and 2.5× lower tLP than OIDN per-frame.
Source: §3.5 · p. 4Takeaways & Limitations
DAGS provides controllable reconstruction and lower perceptual flicker than the evaluated baselines under a matched input budget, though the comparison is not matched compute.
Source: §3.5 · p. 4Training uses indoor scenes, fidelity can drop far from that distribution, and fast camera motion or inconsistent appearance inputs can leave residual flicker.
Source: §4 · p. 5
Abstract
from arXiv · showhide
Diffusion transformers (DiTs) generate high-fidelity images from text and image conditions, but their outputs carry large variance and their faithfulness to a desired target depends heavily on how the condition is supplied. We present DAGS, a lightweight, attention-free, disentangled appearance and geometry conditioning scheme that steers a frozen image DiT to produce high-fidelity, highly faithful, and independently controllable renders. Two small convolutional encoders compute conditioning features once per frame and inject them as a learned, per-layer, element-wise residual into the image tokens, avoiding the quadratic cost of stacking conditions through attention. Because control and temporal handling live outside the frozen backbone, we retain its vast pretrained prior and eliminate backbone-overfitting risk. We further add a small recurrent lighting stabilizer and a training-free temporal guidance term that, coupled with our conditioning, elevate a per-frame image model into a streaming renderer. DAGS produces controllable, high-quality renders at a fraction of the compute of path tracing; it is not real-time, trading compute for controllability and quality. On a matched 1-spp + G-buffer input, per-frame DAGS reconstructs +8.6 dB / +10.1 dB PSNR over the real-time denoiser Intel OIDN and the diffusion renderer RGB<->X while being 2.5-8x more temporally stable perceptually (temporal-LPIPS flicker).
1 Introduction
DAGS addresses the difficulty of precisely controlling diffusion transformers by separating geometry and appearance control. It steers a frozen DiT with lightweight per-frame encoders and handles temporal stabilization outside the backbone.
1 Introduction
- Existing control paradigms use attention whose cost scales quadratically with token count or require repeated coarse correction; ControlNet duplicates about 50% of backbone parameters.The introduction also notes that semantic features are not the metric geometry needed by a renderer.Source: §1 · p. 2
- Splitting geometry and appearance control improves fidelity and yields independent, predictable control, addressing the structure–color trade-off attributed to a single entangled conditioning stream.With entangled control, the model may favor one cue over the other.Source: §1 · p. 2
- GateControl independently supports additive, no-cross-attention conditioning on a linear-attention backbone, while DAGS targets disentangled control and temporally stabilized rendering on a full-attention model.DAGS uses Flux.2 and 1-spp plus G-buffers; GateControl uses SANA.Source: §1 · p. 2
- DAGS uses fewer than 10 M trainable parameters to steer a frozen 4 B DiT with once-per-frame, attention-free CNN residuals, disentangled geometry and appearance control, and temporal stabilization outside the backbone.The design avoids token concatenation while preserving the pretrained prior.Source: §1 · p. 2
2 Method
DAGS conditions a frozen DiT on per-frame geometry and appearance features, while a recurrent lighting stabilizer and optional inference-time guidance address temporal flicker. Its flow starts from a relit prior rather than noise.
2 Method
- Given a 1-spp frame, G-buffers, and a coarse relit prior, DAGS predicts a rendered frame using frozen DiT, text encoder, and VAE components; test-time G-buffers are ground truth.The geometry and appearance encoders supply token-aligned features to the image stream.Source: §2 · p. 2
2.1 Geometry encoder
- Depth and normals form geometry tokens that preserve edges, occlusion boundaries, and surface-orientation cues; dropout helps the compact encoder handle missing or noisy buffers.Because these buffers are deterministic per frame, the geometry stream contributes comparatively little temporal inconsistency.Source: §2.1 · p. 2
2.2 Appearance encoder
- The appearance encoder combines albedo, raw 1-spp RGB, and the relit prior: raw input retains high-frequency lighting, while albedo adds stable surface-boundary cues.The raw 1-spp input is left unconstrained, so residual specular flicker can remain.Source: §2.2 · p. 2
2.3 Recurrent lighting stabilizer
- A small recurrent U-Net creates a facetized relit prior from the same frame and G-buffers, then fuses warped prior estimates with current illumination rather than material color.Validity masks route disoccluded pixels to the spatial estimate; facetization suppresses Monte Carlo speckle while preserving coarse layout.Source: §2.3 · p. 2
2.4 Conditioning injection
- Once-per-frame geometry and appearance features are injected into image tokens across DiT layers using learned per-stream scalars; text tokens and the text encoder remain untouched and frozen.The encoders process depth, normals, albedo, raw 1-spp RGB, and the relit prior.Source: §2.4 · p. 2, Figure 2 · p. 3
2.5 Flow source
- Initializing the flow from the relit prior instead of Gaussian noise shortens the transport path; this choice beats noise on LPIPS in all nine scenes and on PSNR in seven of nine, by up to +8.1 dB.The same relit prior also conditions the appearance encoder.Source: §2.5 · p. 3
2.6 Training-free temporal guidance
- At inference, guidance nudges the decoded estimate toward a motion-warped previous frame in valid regions, operates only on low-noise steps, and adds no trainable weights.It reduces flicker on low or moderate motion, but is weakened or disabled when fast camera motion makes reprojection unreliable.Source: §2.6 · p. 3
3 Experiments
Experiments assess two-stream control, flow-source choice, temporal stabilization, and matched-input reconstruction against OIDN and RGB↔X. DAGS combines geometry and appearance control, and its temporal mechanisms reduce flicker, with gains depending on motion.
3 Experiments
- The experiments report image quality, color, geometry, and temporal metrics, testing two-stream control, independent edits, flow sources, stabilization, and OIDN/RGB↔X baselines.The baseline comparison uses the same 1-spp and G-buffer input budget.Source: §3 · p. 3, §3.5 · p. 4
3.1 Geometry versus appearance
- Geometry-only and appearance-only controls specialize in depth and color respectively, while their combination achieves best or tied MS-SSIM, LPIPS, and geometry in all nine scenes and best PSNR in 8 of 9.Adding albedo to appearance-only control reduces inferred depth error in 7 of 9 scenes, often by half, while Lab ΔE stays flat.Source: §3.1 · p. 3
3.2 Disentanglement
- Lighting edits track relit targets with hue-response ≈0.93 and hue-match ≈0.85 across eight scenes, while geometry edits shift inferred depth toward donor geometry without materially changing color.For geometry edits, Lab ΔE spread is ≤0.73 and hue slope ≤0.06 in 8 of 9 scenes; donor-following is strongest on three high-motion scenes (f_depth = 0.84/0.81/0.71).Source: §3.2 · p. 4
- After removing the shared positional subspace, cross-branch content similarity is small (CKA ≤0.32) versus within-branch structure (0.95–0.99).The authors characterize the branches as disentangled up to shared position, with small residual content overlap.Source: §3.2 · p. 4
3.3 Flow source
- Relit-flow has the best LPIPS in all nine scenes and best or near-best PSNR in 7 of 9, averaging +2.0 dB over noise.Albedo-flow can beat noise on easy interiors but falls up to −5.0 dB on hard cases.Source: §3.3 · p. 4
3.4 Temporal stabilization
- Enabling P and G reduces mean Ewarp from 3.42 to 2.52 and mean tLP from 0.0115 to 0.0053, a 2.2× drop in perceptual flicker.P alone cuts mean tLP by 41%; P and G have additive effects.Source: §3.4 · p. 4
- Temporal gains depend on motion: Ewarp falls by −77% on static content and −31% to −41% on moderate motion, while gains vanish at fast camera motion.For fast motion, the authors disable P and use only low-λ G or per-frame inference.Source: §3.4 · p. 4
3.5 Comparison
- DAGS wins reconstruction on 9 of 9 scenes versus both baselines (+8.6 dB over OIDN, +10.1 dB over RGB↔X, ∼2–3× lower LPIPS) and perceptual stability (tLP) on 9 of 9 (2.5× lower than OIDN per-frame, 5.3× stabilized).Figure 3 shows DAGS (G+P) visually closest to GT on detail and color across three held-out scenes.Source: §3.5 · p. 4, Figure 3 · p. 5
4 Limitations and future work
DAGS is limited by its indoor-only training distribution, inconsistent appearance inputs, and motion regimes where reprojection becomes unreliable. Heavy disocclusion also constrains its external temporal handling; the authors identify broader training data, richer appearance conditioning, and temporal feedback during sampling as future directions.
4 Limitations and future work
- Indoor-only training limits fidelity far from the training distribution, motivating a larger and more diverse corpus.The evaluation includes held-out and out-of-distribution scenes.Source: §4 · p. 5
- Frame-to-frame inconsistent appearance inputs leave high-frequency detail and specular highlights vulnerable to residual flicker.The authors propose detail- and specular-aware appearance conditioning.Source: §4 · p. 5
- Under fast camera motion, low reprojection validity means guidance cannot outperform per-frame inference, and the recurrent stabilizer must be disabled to avoid ghosting.Source: §4 · p. 5
- Heavy disocclusion breaks reprojection relied on by the external temporal handling; the authors suggest feeding previous decoded frames into diffusion sampling while keeping the backbone frozen.Source: §4 · p. 5