PaperlayerView exact revision on arXiv

Source-linked AI summary

NeuralSRender: G-Buffer Guided Video Diffusion for Sim-to-Real Rendering

Yipeng Lu, Youqi Liao, Zhenyu Wang, Yuning Peng, Zhen Dong

arXiv:2610.29200v1cs.CV

TL;DR

Real driving data is costly to collect and annotate, while synthetic videos offer scale but have an appearance gap. NeuralSRender translates synthetic videos using G-buffer conditioning and staged adaptation; the authors report improved realism with competitive geometric consistency and stronger real-world perception results from translated data.

  • Problem

    Real-world driving data is costly to collect and annotate, and simulator videos have an appearance gap that can limit models’ real-world generalization.

    Source: §I · p. 1

  • Method

    NeuralSRender uses a lightweight projector to condition a pretrained video diffusion model on G-buffers, with staged training on synthetic and real videos.

    Source: §I · pp. 1–2, §IV · p. 3

  • Results

    NeuralSRender achieves the best FVD and VLM-BT scores on GenReal-City and GenReal-CARLA while maintaining competitive CLIP-T scores and NDE.

    Source: B. Comparison Results · p. 6

  • Takeaways & Limitations

    The authors conclude that NeuralSRender enables high-quality, scalable video data generation for autonomous-driving models.

    Source: §VI · p. 7

Abstract

from arXiv · show

The scarcity of high-quality annotated visual data remains a critical bottleneck for autonomous-driving perception. Simulators provide training data at scale, but the limited realism of synthetic videos compromises perception performance in real-world applications. To address this limitation, we propose NeuralSRender, a sim-to-real video translation framework that enhances the photorealism of synthetic videos while maintaining geometric consistency with the source scenes. Built on a pretrained video diffusion transformer (DiT), NeuralSRender uses a lightweight projector to inject geometric and material priors from G-buffers, and LoRA enables parameter-efficient adaptation. The model is trained in two stages: it first learns G-buffer conditioning from synthetic videos with accurate G-buffers, then adapts to real-world appearance using real videos with estimated G-buffers. During the second stage, an additional LoRA module is trained while the projector and first-stage LoRA remain frozen to reduce interference from estimation noise. NeuralSRender introduces only 1.45\% parameter overhead and can be trained on consumer-grade GPUs with 24 GB of memory. To support training and evaluation, we introduce GenReal, a large-scale driving video dataset comprising synthetic and real-world videos with corresponding G-buffers. Experiments demonstrate improved photorealism with competitive geometric consistency. Training perception models on our translated videos instead of raw simulator renders improves detection $\mathrm{AP}_{50}$ on nuImages from 35.1 to 40.6 and segmentation mIoU on Cityscapes from 32.4 to 40.8. Our code, data, and model weights are available at: https://github.com/WHU-USI3DV/NeuralSRender.

I. INTRODUCTION

Autonomous-driving perception needs diverse annotated data, but real-world collection is costly and simulators have an appearance gap. NeuralSRender uses G-buffer conditioning and staged adaptation to translate synthetic videos, alongside the GenReal dataset.

I. INTRODUCTION

  • Real-world driving data is costly and labor-intensive to collect and annotate, while routine collection misses rare, safety-critical scenarios.Source: §I · p. 1
  • Synthetic videos scale data generation and provide automatic annotations, but their appearance gap can limit real-world generalization.The passage attributes the gap to costly-to-scale textures and materials and simplified light-transport models in real-time rendering.Source: §I · p. 1
  • The authors report improved visual realism with competitive geometric consistency, and better real-world detection and segmentation performance from translated synthetic data than from other synthetic data.Source: §I · p. 1
  • NeuralSRender combines dense geometric and material conditioning with pretrained video-generation priors, adding only 1.45% parameter overhead relative to the backbone.A lightweight projector injects the G-buffer features into a pretrained video diffusion model.Source: §I · pp. 1–2
  • The method first learns conditioning from accurate synthetic G-buffers, then adapts appearance on real videos while freezing learned conditioning components to limit interference from noisy estimates.The second-stage adaptation trains additional LoRA modules while the projector and Stage-I LoRA remain frozen.Source: §I · pp. 1–2
  • GenReal spans two synthetic domains and one real-world domain, with G-buffer representations and text prompts for training and cross-domain evaluation.Source: §I · pp. 1–2

II. RELATED WORK

Prior work offers video control, sim-to-real translation, and neural rendering, but the cited approaches leave gaps in dense geometry and material guidance or broad appearance adaptation. NeuralSRender combines G-buffers and text guidance for outdoor sim-to-real video translation.

a) Controllable Video Generation

  • Controllable video methods inject semantic, structural, or motion cues, while NeuralSRender uses multi-attribute G-buffers to provide dense geometric and material guidance.Source: §II · p. 2

b) Sim-to-Real Translation

  • Image-based translation does not model temporal consistency, while semantic-map video synthesis lacks explicit use of simulators’ dense geometry and material information.Source: §II · p. 2

c) Neural Rendering

  • Relighting methods mainly target lighting changes, and DiffusionRenderer focuses on object-centric relighting without text guidance; NeuralSRender combines dense G-buffers with text-guided adaptation for outdoor scenes.Source: §II · pp. 2–3

III. GENREAL DATASET

GenReal addresses the scarcity of datasets combining realistic driving video and dense rendering attributes. It brings together synthetic and real-world samples, while the accompanying method uses G-buffers and text prompts for staged sim-to-real video translation.

III. GENREAL DATASET

  • GenReal combines GenReal-City, GenReal-CARLA, and GenReal-DV, with RGB video, basecolor, depth, normal, metallic, and roughness sequences, plus a text prompt per annotated sample.Source: §III · p. 3
  • GenReal-City and GenReal-CARLA use simulator-rendered RGB sequences with corresponding G-buffers, and their differences in layouts, assets, and rendering styles increase synthetic-data diversity.Source: §III · p. 3
  • GenReal-DV uses real driving clips with attributes estimated by DiffusionRenderer; severe visual defects are removed, while mild artifacts are retained.The estimated attributes are noisy, unlike simulator-exported G-buffers.Source: §III · p. 3
  • Video prompts combine human labels for low-level visual attributes with Qwen3-VL [34] descriptions of scene semantics, object layout, road conditions, and background.Source: §III · p. 3

IV. METHOD

  • NeuralSRender generates driving videos from G-buffers and text prompts using a lightweight projector and a pretrained video diffusion model.Its two-stage training learns conditioning from accurate synthetic G-buffers before adapting to real-world appearance while keeping learned conditioning components fixed.Source: §IV · p. 3

A. G-buffer Condition Fusion

NeuralSRender encodes aligned G-buffer attributes into the video model’s latent space and fuses them with the noisy video latent. A lightweight projector integrates these features so generation can follow geometric and material constraints alongside text prompts.

A. G-buffer Condition Fusion

  • Depth and normals describe geometry, while basecolor, metallicity, and roughness describe material appearance across five spatially and temporally aligned G-buffer sequences.The attributes correspond to the RGB video.Source: A. G-buffer Condition Fusion · p. 3
  • The generation goal is a photorealistic video that preserves G-buffer geometry and material constraints while following text-specified appearance such as weather, illumination, and time of day.Source: A. G-buffer Condition Fusion · p. 3
  • Each G-buffer attribute sequence is independently encoded by the frozen video VAE encoder into the video model’s spatiotemporal latent space.This maps attributes to the video latent’s spatial and temporal resolution while preserving their individual information.Source: A. G-buffer Condition Fusion · p. 3
  • A lightweight 3D convolutional projector integrates the encoded G-buffer features with the noisy video latent and maps the fused representation back to the original latent channel dimension.The projector captures spatiotemporal interactions and relations among attributes before the result is passed to the DiT with the timestep and text prompt.Source: A. G-buffer Condition Fusion · p. 4

B. Two-Stage Sim-to-Real Adaptation

NeuralSRender separates learning reliable G-buffer conditioning from adapting appearance to real-world video. Accurate synthetic G-buffers train the projector and first LoRA; real videos with estimated G-buffers then train a new LoRA while the learned conditioning components remain frozen.

B. Two-Stage Sim-to-Real Adaptation

  • Real-world G-buffers can contain noise and inverse-rendering artifacts, so directly training on them may teach the projector estimation errors instead of reliable attribute relations.Source: B. Two-Stage Sim-to-Real Adaptation · p. 4
  • Stage II freezes the Stage-I projector and LoRA, then trains an additional LoRA on real videos with estimated G-buffers to adapt lighting, textures, materials, and other appearance statistics.The pretrained DiT weights remain frozen throughout training.Source: B. Two-Stage Sim-to-Real Adaptation · pp. 4–5
  • Stage I jointly trains the G-buffer projector and a LoRA module on synthetic videos with accurate G-buffers to learn physical and spatial correspondences between attributes and generated content.The stage learns how physical G-buffer priors map into visual generation.Source: B. Two-Stage Sim-to-Real Adaptation · p. 4
  • The qualitative comparison on GenReal-City and GenReal-CARLA presents the approach as combining G-buffer geometry and physical priors with text prompts to achieve photorealistic synthesis while maintaining structural integrity.Source: Figure 3 · p. 5

A. Experimental Setup

The experiments use a pretrained video diffusion model and staged LoRA training, then assess video quality, text alignment, photorealism, geometric consistency, and efficiency. Baselines share test clips and output resolution, with prompts supplied to text-conditioned methods.

A. Experimental Setup

  • NeuralSRender is built on Wan2.2-TI2V-5B, with AdamW, weight decay 1 × 10−4, and LoRA rank 32 in both stages.Source: A. Experimental Setup · p. 5
  • The qualitative comparison contrasts depth, edge, and basecolor controls with two-stage-without-freeze and single-stage training against the full model.The caption reports that the full model best preserves geometry, markings, textures, lighting, and shadows.Source: Figure 4 · p. 6
  • Stage I trains the projector and Stage-I LoRA on GenReal-City for 6 epochs; Stage II freezes the projector and trains a new LoRA on GenReal-DV for 2 epochs.Training uses batch size 4 and takes approximately five days on four RTX 4090 GPUs.Source: A. Experimental Setup · p. 6
  • Generation is evaluated with FVD, CLIP-T, and VLM-BT, geometric consistency with NDE, and efficiency with peak VRAM and FPS at 1280 × 704 × 81.VLM-BT reflects relative preferences within each independently fitted comparison set.Source: A. Experimental Setup · p. 6
  • The baselines are DiffusionRenderer, TC-Light, and edge-controlled Cosmos, tested on the same clips and output resolution with identical prompts for text-conditioned methods.DiffusionRenderer is evaluated without text conditioning.Source: A. Experimental Setup · p. 6

B. Comparison Results

NeuralSRender improves visual realism while retaining competitive text alignment and geometric consistency. It outperforms the compared methods on FVD and VLM-BT across both synthetic test sets.

B. Comparison Results

  • NeuralSRender achieves the best FVD and VLM-BT scores on both GenReal-City and GenReal-CARLA test sets.The reported quantitative results are in Table II.Source: B. Comparison Results · p. 6
  • It maintains competitive CLIP-T scores and NDE without substantially sacrificing text alignment or geometric consistency.Qualitative comparisons also show a favorable balance between visual realism and structural consistency.Source: B. Comparison Results · p. 6
  • DiffusionRenderer closely follows G-buffer cues but retains an artificial appearance, while TC-Light mainly adjusts global color and leaves original textures and shadows.Cosmos produces visually appealing results, but edge conditioning can lead to geometric deviations and artifacts.Source: B. Comparison Results · p. 6

C. Ablation Studies

The ablations compare individual depth, edge, and basecolor controls with full G-buffer conditioning, and test alternatives to the two-stage training strategy. Full conditioning and freezing the projector during real-world adaptation yield the strongest reported results.

C. Ablation Studies

  • Two-Stage w/o Freeze keeps the projector trainable during real-world adaptation, while Single-Stage Training mixes synthetic and real samples in one stage.The control-signal variants use only basecolor, depth, or edge, and the compared experiments use the same data and number of training steps.Source: C. Ablation Studies · pp. 6–7
  • Full G-buffer conditioning combines complementary geometry and material cues, improving structural preservation and appearance over single controls.Depth struggles with near and distant regions and cannot constrain road markings or colors; edges lack color and texture, while basecolor alone can merge geometrically distinct regions.Source: C. Ablation Studies · p. 7
  • The full model achieves the best FVD, NDE, and VLM-BT scores, with competitive CLIP-T scores and strong geometric and visual quality.It freezes the G-buffer projector during real-world adaptation while LoRA adapts the model to real-world appearance.Source: C. Ablation Studies · p. 7
  • Without freezing, noisy real-data G-buffers partly destroy learned attribute relations and reduce realistic lighting and shadows, despite preserving overall geometry.Single-stage training is closest to the full model, but noisy real data can still distort geometry and semantics, including car orientation and ground markings.Source: C. Ablation Studies · p. 7

D. Sim-to-Real Transfer for Autonomous Driving

Perception models trained on NeuralSRender-translated videos perform better on unseen real-world datasets than models trained on raw CARLA renders. The comparison is conducted without target-domain fine-tuning and includes object detection and semantic segmentation.

D. Sim-to-Real Transfer for Autonomous Driving

VI. CONCLUSION

The paper presents NeuralSRender for G-buffer-guided sim-to-real video translation and introduces GenReal, a dataset of synthetic and real-world videos with corresponding G-buffers. It concludes that the approach improves visual realism with competitive geometric consistency and supports scalable video-data generation for autonomous-driving models.

VI. CONCLUSION

  • NeuralSRender is a parameter-efficient framework for G-buffer-guided sim-to-real video translation.Its two-stage training separates reliable conditioning learning from real-world appearance adaptation.Source: §VI · p. 7
  • GenReal contains synthetic and real-world driving videos with corresponding G-buffers.Source: §VI · p. 7
  • Experiments show improved visual realism with competitive geometric consistency, and the paper frames the work as enabling scalable video-data generation for autonomous-driving models.Source: §VI · p. 7

Related papers

Loading 2610.29200v1…