PaperlayerView exact revision on arXiv

Source-linked AI summary

FrameDiffuser: G-Buffer-Conditioned Diffusion for Neural Forward Frame Rendering

Ole Beisswenger, Jan-Niklas Dihlmann, Hendrik P. A. Lensch

arXiv:2512.16670v1cs.CVcs.GR

TL;DR

Interactive applications need frame-by-frame rendering that can respond to user input, a gap left by independent image models and complete-sequence video models. FrameDiffuser combines G-buffer structural guidance with previous-frame temporal guidance and environment-specific training. The authors report stable generation over extended sequences and identify inference speed and per-style model requirements as limitations.

  • Problem

    Interactive rendering needs user-driven frame-by-frame generation, which prior single-image and complete-sequence video approaches do not provide.

    Source: §1 · p. 2, §5 · p. 8

  • Method

    FrameDiffuser combines G-buffer conditioning and previous-frame conditioning in an autoregressive framework, using environment-specific models and three-stage self-conditioning training.

    Source: §1 · p. 2, §5 · p. 8

  • Results

    The authors report that environment-specific specialization improves consistency within specific domains and that self-conditioning supports temporal consistency across extended sequences.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The approach prioritizes consistent rendering within individual visual domains over moderate generalization across environments.

    Source: §5 · p. 8

    Separate models are required for different visual styles, and inference currently runs at approximately 1 frame per second on consumer hardware.

    Source: §5.1 · p. 8

Abstract

from arXiv · show

Neural rendering for interactive applications requires translating geometric and material properties (G-buffer) to photorealistic images with realistic lighting on a frame-by-frame basis. While recent diffusion-based approaches show promise for G-buffer-conditioned image synthesis, they face critical limitations: single-image models like RGBX generate frames independently without temporal consistency, while video models like DiffusionRenderer are too computationally expensive for most consumer gaming sets ups and require complete sequences upfront, making them unsuitable for interactive applications where future frames depend on user input. We introduce FrameDiffuser, an autoregressive neural rendering framework that generates temporally consistent, photorealistic frames by conditioning on G-buffer data and the models own previous output. After an initial frame, FrameDiffuser operates purely on incoming G-buffer data, comprising geometry, materials, and surface properties, while using its previously generated frame for temporal guidance, maintaining stable, temporal consistent generation over hundreds to thousands of frames. Our dual-conditioning architecture combines ControlNet for structural guidance with ControlLoRA for temporal coherence. A three-stage training strategy enables stable autoregressive generation. We specialize our model to individual environments, prioritizing consistency and inference speed over broad generalization, demonstrating that environment-specific training achieves superior photorealistic quality with accurate lighting, shadows, and reflections compared to generalized approaches.

1. Introduction

Interactive neural rendering must generate frames in response to user input, but prior approaches either generate images independently or require complete video sequences. FrameDiffuser combines autoregressive G-buffer-conditioned generation with environment-specific training for temporally consistent rendering.

1. Introduction

  • Single-image models generate frames independently, while video models require complete sequences, leaving neither suited to user-driven frame-by-frame rendering.The paper also identifies inconsistent geometry and imprecise, laggy control in attempts to simulate whole interactive environments.Source: §1 · p. 2
  • The approach specializes models to individual visual domains because consistent art style and lighting within a game may favor specialization over broad generalization.The paper frames this as an exploration inspired by environment-specific training in EST-GAN.Source: §1 · p. 2
  • FrameDiffuser generates photorealistic frames from G-buffer data and its own previous output, maintaining stable generation over hundreds to thousands of frames.The framework is trained for six Unreal Engine 5 environments.Source: §1 · p. 2
  • The contributions include separate structural and temporal conditioning, three-stage self-conditioning training, irradiance-based lighting guidance, and environment-specific specialization.The authors report that progressive training is critical for preventing error accumulation in autoregressive generation.Source: §1 · p. 2
  • FrameDiffuser is positioned as an augmentation to traditional rendering that preserves artist control over world building rather than replacing the rendering pipeline.The paper investigates this approach within frame-by-frame generation and domain specialization.Source: §1 · p. 2

2. Related Work

Related work spans diffusion-based image and video generation, G-buffer-conditioned rendering, and techniques for interactive autoregressive generation. FrameDiffuser combines these lines by pairing G-buffer conditioning with autoregression and environment-specific models.

Image Generation

  • Diffusion models reverse a gradual noising process, while Stable Diffusion improves efficiency through compressed latent-space generation.Earlier GAN approaches are described as susceptible to training instability and mode collapse.Source: §2 · p. 2

Autoregressive Generation for Interactive Applications

  • Interactive autoregression must address both user-driven frame-by-frame input and drift from training on perfect inputs but inferring on generated outputs.Self Forcing [17] and related work [2] [4] train on model-generated outputs to bridge that distribution gap.Source: §2 · p. 3

G-Buffer-Based Neural Rendering

  • EST-GAN paired G-buffer conditioning with environment-specific training, but its GAN-based approach lacked temporal consistency and faced GAN limitations.Its reported visual-quality gains were over semantic maps alone.Source: §2 · p. 3
  • G-buffer-conditioned image models use geometric and material data for single-image synthesis, whereas DiffusionRenderer generates temporally coherent complete video sequences.RGB↔X [32] focuses on individual images; Xue et al. [31] report better G-buffer conditioning with ControlNet than direct channel concatenation.Source: §2 · p. 4

Summary

  • FrameDiffuser addresses the gap between independent image generation and complete-sequence video generation by combining G-buffer conditioning with autoregression and specialized environment models.The related-work discussion describes RGB↔X [32] as lacking temporal consistency and DiffusionRenderer as requiring entire sequences.Source: §2 · p. 4

3. Method

FrameDiffuser uses G-buffer data for structural guidance and previous-frame information for temporal guidance, then trains autoregressively to improve robustness. Additional temporal-coherence mechanisms address offset sampling, sky regions, and noisy previous frames.

3. Method

  • The framework separates G-buffer-based structural guidance from previous-frame temporal guidance to support autoregressive neural rendering.The architecture builds on Stable Diffusion 1.5, with ControlNet and ControlLoRA as complementary conditioning mechanisms.Source: §3 · p. 4, §3.1 · p. 4

3.1. Architecture

  • ControlNet processes nine G-buffer channels plus irradiance, while ControlLoRA conditions on the previous frame encoded in VAE latent space.The previous frame and corresponding irradiance initialize generation, after which subsequent frames are synthesized autoregressively.Source: §3.1 · p. 4
  • Irradiance derived from the previous generated frame and basecolor supplies ControlNet with temporal lighting cues, including guidance about shadow and specular locations.The irradiance map is normalized for network input and kept distinct from ControlLoRA’s temporal appearance role.Source: §3.1 · p. 4

3.2. Training Strategy

  • Training progresses from G-buffer translation with black irradiance, to joint temporal conditioning, to self-conditioning on frames generated by current model weights.The final stage addresses the mismatch between perfect training data and autoregressive deployment, making the model more resilient to its own artifacts.Source: §3.2 · pp. 4–5

3.3. Temporal Coherence Mechanisms

  • Qualitative results show autoregressive outputs and G-buffer channels across frames, with the figure captions reporting long-term stability through frame 4000 and more detailed lighting than X→RGB.The comparison covers Downtown West and Hillside Sample; X→RGB is described as applying more uniform lighting.Source: Figure 3 · p. 5, Figure 4 · p. 6
  • Temporal offsets are sampled from −2 to 2 with adjacent frames weighted more heavily to reduce overfitting to single-frame transitions.Sky regions lack geometric G-buffer guidance, so adaptive masking injects sky information into basecolor; τ = 10/255 identifies black pixels in basecolor and depth.Source: §3.3 · p. 5
  • Noise is added to the previous RGB frame before irradiance computation, and both conditioning paths use that same noisy frame to encourage greater G-buffer use.The noise scale is sampled uniformly from 0.0 to 0.2.Source: §3.3 · p. 6

4. Experiments and Results

FrameDiffuser is evaluated on autoregressive frame generation across six environment-specific validation sets, with analyses of training mechanisms, lighting, scene editing, and baseline comparisons. The results report improved structural and perceptual metrics over X→RGB and stable generation with self-conditioning and noise injection, alongside specific long-sequence failure conditions.

4.1. Experimental Setup

  • The evaluation uses separate models for six Unreal Engine 5 environments, held-out validation splits, and SSIM, PSNR, and LPIPS metrics.X→RGB is the neural-rendering baseline described for the setup.Source: §4.1 · p. 6

4.2. Quantitative Results

  • FrameDiffuser achieves higher SSIM and lower LPIPS than X→RGB on held-out validation sets from its training environments.The comparison is averaged across validation sets from FrameDiffuser’s training distribution.Source: §4.2 · p. 6, Table 1 · p. 6

4.3.1. Self-Conditioning and Noise Injection

  • During 3000+ frames of pure autoregressive generation, metrics degrade in very dark rooms and after frame 3000 when movement and G-buffer changes become minimal.The passage attributes the dark-room issue to bias toward lit areas and the later degradation to error accumulation.Source: Figure 5 · p. 7
  • On the 589-frame MidRange sequence, self-conditioning and noise injection improve PSNR by 5.87 dB, SSIM by 40.6%, and LPIPS by 41.3%.Without these mechanisms, quality degrades within five frames; with them, it remains stable over extended sequences.Source: §4.3.1 · p. 7, Table 2 · p. 7, Figure 6 · p. 7

4.3.2. Irradiance Conditioning

  • On 900 City Park validation frames, adding irradiance yields small but consistent improvements across all metrics, while G-buffer alone provides strong guidance.The ablation compares models trained with and without the irradiance channel.Source: §4.3.2 · p. 7, Table 3 · p. 7

4.4. Qualitative Results

  • Across frames including frame 4000, qualitative examples show temporal consistency, geometric and material fidelity, and realistic shadows, reflections, and lighting.The irradiance channel is computed from the previous model output and previous base-color input.Source: §4.4 · p. 7

4.5. Scene Editing Capabilities

  • When G-buffer edits add objects, FrameDiffuser synthesizes corresponding lighting, shading, and cast shadows, including complex patterns on a large statue.The described workflow leaves artists in control of scene composition while the model handles lighting synthesis.Source: §4.5 · p. 8, Figure 7 · p. 8

4.6. Comparison with Baselines

  • Compared with X→RGB, FrameDiffuser produces richer lighting and more temporally consistent frames, while X→RGB appears flat and changes lighting inconsistently.The comparison describes disappearing door-frame shadows and shifting floor-lighting patterns in X→RGB outputs.Source: §4.6 · p. 8

5. Conclusion

FrameDiffuser uses ControlNet for geometric guidance and ControlLoRA for temporal coherence in interactive frame-by-frame neural rendering. Its environment-specific approach prioritizes consistency over generalization, with current inference speed and per-style model requirements limiting deployment scope.

5. Conclusion

  • FrameDiffuser combines ControlNet geometric guidance with ControlLoRA temporal coherence for frame-by-frame rendering that accommodates user-driven frames.The conclusion contrasts this with single-image models’ lack of temporal consistency and video models’ inability to accommodate user-driven frame generation.Source: §5 · p. 8
  • Three-stage training with self-conditioning is described as essential for extended temporal consistency, while environment-specific models prioritize within-domain consistency over broad generalization.The approach requires separate models for different visual styles and reaches approximately 1 frame per second on consumer hardware.Source: §5 · p. 8, §5.1 · p. 8

5.1. Limitations and Future Work

  • Future work includes combining video diffusion with G-buffer conditioning, integrating 3D scene methods, and exploring prompt-based scene-style changes.Source: §5.1 · p. 8

Rendering

Supplementary videos are presented as a useful medium for evaluating FrameDiffuser’s temporal results. They include a Hillside sequence with dark-room mismatches, model comparisons, and a demonstration of VAE-related flicker.

Rendering

  • The supplementary videos cover autoregressive Hillside generation, comparisons with DiffusionRenderer, and VAE flicker in a metropolitan scene.The authors encourage viewing the videos because the results are temporal in nature.Source: Rendering · p. 11
  • In the Hillside sequence, predictions in completely dark rooms remain far from ground truth despite approximating reasonable illumination.The text says these mismatches are also visible in the temporal stability analysis figure.Source: Rendering · p. 11
  • The City Sample video attributes frame-to-frame inconsistencies, especially in high-spatial-frequency scenes, primarily to the VAE encoder-decoder cycle.Source: Rendering · p. 11

A. DiffusionRenderer Comparison

FrameDiffuser generates each frame from the current G-buffer and the previously generated frame, whereas DiffusionRenderer processes complete sequences. A 24-frame comparison reports closer alignment to ground-truth lighting and atmospheric effects for FrameDiffuser, while DiffusionRenderer shows inconsistent lighting and reduced detail.

A. DiffusionRenderer Comparison

  • DiffusionRenderer uses complete sequences during generation, while FrameDiffuser uses the current G-buffer and previous generated frame, matching interactive settings where future frames depend on user input.The quantitative comparison uses 24 frames, the maximum sequence length DiffusionRenderer supports, while FrameDiffuser can generate arbitrarily long autoregressive sequences.Source: A. DiffusionRenderer Comparison · p. 11
  • Across 24-frame sequences averaged over six environments, FrameDiffuser stays closer to ground-truth lighting and atmospheric effects than DiffusionRenderer.The qualitative comparison covers two sequences from City Park and Hillside Sample Project.Source: A. DiffusionRenderer Comparison · p. 11, Table 4 · p. 11

B. Out-of-Distribution Generalization

The out-of-distribution test evaluates a Downtown West-trained model on City Sample, whose architecture and lighting differ. FrameDiffuser performs well despite the mismatch, but extended autoregressive generation increasingly shifts toward the training environment’s warm style.

B. Out-of-Distribution Generalization

C. Temporal Flicker Analysis

The reported temporal flicker is associated with the Stable Diffusion 1.5 VAE, especially in high-spatial-frequency scenes. Addressing it with a specialized VAE is outside this work’s frame-generation scope.

C. Temporal Flicker Analysis

  • The Stable Diffusion 1.5 VAE introduces temporal flicker, especially in scenes with high spatial frequencies.The paper illustrates this with a metropolitan city example.Source: C. Temporal Flicker Analysis · p. 12
  • The authors believe a different VAE could substantially reduce or eliminate the flicker.They mention fine-tuned VAE decoders or VAEs incorporating previous-frame context as possible solutions.Source: C. Temporal Flicker Analysis · p. 12
  • Training or fine-tuning a specialized VAE is outside the work’s scope, which focuses on the frame-generation pipeline.Source: C. Temporal Flicker Analysis · p. 12

D. GBufferDiffuser: Inverse Rendering

GBufferDiffuser reconstructs G-buffer components from rendered images using separately specialized models, achieving stronger inverse-rendering results on its target environment than the generalist baseline. The style-transfer experiment uses a stylized first frame to guide autoregressive generation, but the effect fades as outputs return toward the training distribution.

D. GBufferDiffuser: Inverse Rendering

E. Style Transfer

  • For style transfer, an image-to-image transformation of the original frame becomes the previous-frame input for autoregressive generation, alongside a prompt for the new style.The example uses a burning-scene prompt.Source: E. Style Transfer · p. 13
  • In the burning-style example, fire effects persist but gradually diminish as generation reverts to the training distribution.The stylized frame conditions subsequent autoregressive output.Source: E. Style Transfer · p. 13, Figure 11 · p. 13
  • Style-transfer capability is limited because specialization on a fixed prompt-environment combination reduces response to alternative text conditioning.Source: E. Style Transfer · p. 13

F. Implementation Details

The implementation uses consecutive frame pairs with corresponding G-buffers, with validation sequences spanning short-term evaluation through long-term autoregressive stability analysis. Training uses a three-stage schedule, while inference remains approximately one frame per second on an RTX 4090; the authors identify untested optimization options.

F. Implementation Details

  • Validation sequences range from 24 frames to over 3500 frames for short-term and long-term autoregressive stability evaluation.Dataset sets contain consecutive frame pairs with corresponding G-buffers.Source: F. Implementation Details · pp. 13–14
  • Training uses batch size 2 with gradient accumulation of 4, for an effective batch size of 8, and AdamW with weight decay 10^-2 and cosine learning-rate scheduling.Stage 1 trains ControlNet for 40k steps; stages 2 and 3 continue for 10k and 30k steps at reduced learning rates.Source: F. Implementation Details · p. 14
  • Inference runs at approximately 1 frame per second on an RTX 4090 using 10 denoising steps and the DPMSolver scheduler.Training both ControlNet and ControlLoRA requires approximately 23GB VRAM.Source: F. Implementation Details · p. 14
  • Batching, mixed precision, StreamDiffusion, distillation, and NVIDIA DLSS are identified as optimization possibilities not explored in this work.Source: F. Implementation Details · p. 14
Loading 2512.16670v1…