Source-linked AI summary
Generative World Renderer
Zheng-Hui Huang, Zhixiang Wang, Jiaming Tan, Ruihan Yu, Yidan Zhang, Bo Zheng, Yu-Lun Liu, Yung-Yu Chuang, Kaipeng Zhang
TL;DR
Existing synthetic datasets limit real-world bidirectional rendering because they lack realism and temporal coverage. The paper introduces a large-scale AAA-game dataset, capture toolkit, and VLM-based evaluation protocol, and reports improved decomposition, G-buffer-conditioned synthesis, and controllable editing after fine-tuning.
Problem
Existing synthetic datasets provide limited realism and temporal coherence for scaling generative inverse and forward rendering to real-world scenarios.
Method
The paper constructs a continuous AAA-game dataset with synchronized RGB and five G-buffer channels, motion-blur variants, a dual-screen capture pipeline, and VLM evaluation of semantic, spatial, and temporal consistency.
Results
Fine-tuning on the dataset substantially improves robust material decomposition and the fidelity of G-buffer-conditioned video synthesis, with controllable editing demonstrated as an application.
Takeaways & Limitations
The dataset and training recipe provide a foundation for temporally coherent bidirectional rendering and text-guided editing of AAA-game styles from G-buffers.
Abstract
from arXiv · showhide
Scaling generative inverse and forward rendering to real-world scenarios is bottlenecked by the limited realism and temporal coherence of existing synthetic datasets. To bridge this persistent domain gap, we introduce a large-scale, dynamic dataset curated from visually complex AAA games. Using a novel dual-screen stitched capture method, we extracted 4M continuous frames (720p/30 FPS) of synchronized RGB and five G-buffer channels across diverse scenes, visual effects, and environments, including adverse weather and motion-blur variants. This dataset uniquely advances bidirectional rendering: enabling robust in-the-wild geometry and material decomposition, and facilitating high-fidelity G-buffer-guided video generation. Furthermore, to evaluate the real-world performance of inverse rendering without ground truth, we propose a novel VLM-based assessment protocol measuring semantic, spatial, and temporal consistency. Experiments demonstrate that inverse renderers fine-tuned on our data achieve superior cross-dataset generalization and controllable generation, while our VLM evaluation strongly correlates with human judgment. Combined with our toolkit, our forward renderer enables users to edit styles of AAA games from G-buffers using text prompts.
1 Introduction
The paper addresses the data bottleneck limiting bidirectional rendering in real-world scenarios by introducing a large-scale, continuous AAA-game dataset with synchronized RGB and G-buffers. It also provides a capture pipeline, motion-blur variants, VLM evaluation, and evidence of improved decomposition and controllable editing.
- Motivation: Existing synthetic datasets lack the scene complexity, real elements, fine-grained detail, dynamic motion, and long-range temporal dependencies needed for in-the-wild rendering.These limitations contribute to a persistent domain gap in real-world videos.
- Dataset: The dataset contains over 4M 720p/30fps frames from two AAA games, with synchronized RGB and five G-buffer channels across long sequences, environments, and weather conditions.The channels are depth, normals, albedo, metallic, and roughness.
- Bidirectional Rendering: The dataset supports inverse material decomposition and forward G-buffer-conditioned synthesis, including volumetric effects such as fog and rain.Generative models learn a flexible prior that extends beyond rigid geometry.
- Evaluation and Applications: Clean RGB frames paired with synthesized motion-blur variants improve practical coverage, while the VLM protocol evaluates semantic, spatial, and temporal consistency against human preferences.The protocol targets real-world evaluation settings where traditional per-frame metrics are insufficient.
- Data Curation Pipeline: A graphics-API interception and dual-screen stitching pipeline enables scalable acquisition of high-resolution G-buffers without decompilation or asset extraction.The capture strategy records buffers with minimal quality loss.
- Results: Fine-tuning on the dataset improves state-of-the-art models in decomposition and controllable editing, supported by a VLM-based ranking protocol for real-world assessment.The contribution claim covers both rendering performance and evaluation.
2 Related Work
Related work spans forward and inverse rendering, specialized datasets, temporal-consistency methods, and evaluation protocols. Across these areas, prior work provides rendering techniques and synthetic supervision but leaves challenges in real-world G-buffers, long temporal sequences, and holistic evaluation.
- Forward and Inverse Rendering: Forward-rendering research synthesizes images from geometry, materials, and lighting, with path tracing and spatiotemporal sampling enabling interactive rendering of complex dynamic scenes.The related work includes modern rendering pipelines and neural rendering paradigms.
- Forward and Inverse Rendering: Inverse-rendering methods decompose images into geometry, reflectance, and materials using optimization, learning, neural fields, Gaussian splatting, and diffusion-based approaches.These methods support tasks including relighting, material estimation, and video-level PBR extraction.
- Datasets and Game-Based Collection: Existing datasets provide indoor, outdoor, object-centric, material, relighting, and driving resources, but acquiring real-world G-buffers remains challenging and synthetic supervision remains essential.The dataset landscape covers diverse annotations and scene scales without resolving the central capture difficulty.
- Temporal Consistency and Depth Estimation: Temporal-consistency research addresses motion blur and coherent video generation through recurrent networks, video priors, diffusion, feature propagation, and spatiotemporal constraints.Synthetic clean or degraded passes motivate generating blur through frame interpolation.
- Evaluation Protocols: Standard image and video metrics miss cross-buffer consistency, motivating perceptual, video, VLM, preference-learning, and multimodal evaluation approaches.The literature includes VLM-based assessment for semantic quality, faithfulness, compositionality, 3D evaluation, and depiction.
3 Dataset Construction
The dataset construction pipeline captures, validates, annotates, and post-processes synchronized RGB and G-buffer data from diverse game sequences. It combines runtime buffer interception, multi-screen recording, material-channel separation, motion-blur synthesis, and metadata filtering.
- 3.1 G-buffer Interception: ReShade intercepts runtime rendering outputs, while RenderDoc inspection and metadata-based signatures identify valid G-buffer attachments across game-specific layouts.Selected buffers are manually validated and checked by re-rendering RGB frames through deferred shading for pixel-level consistency.
- 3.1 G-buffer Interception: Camera-space normals are reconstructed from depth using inverse projection and finite differences because reliable view-matrix access is unavailable.The capture pipeline can reliably obtain world-space normals but cannot directly convert them to camera space.
- 3.2 Material Capture: Material maps are separated into spatially distinct screen regions to prevent compression noise from coupling metallic and roughness channels.This avoids inter-channel bleeding when packed material channels are recorded through screen capture.
- 3.3 Synchronized Capture: A mosaic canvas and two stitched 2K monitors enable near-lossless, temporally synchronized capture of all six channels at an effective 720p resolution.The pipeline shades target buffers to the screen and records them with hardware-accelerated capture rather than direct per-frame G-buffer export.
- 3.4 Data Collection: Sequences from Cyberpunk 2077 and Black Myth: Wukong use varied traversal strategies to cover urban, natural, indoor, and dynamic viewpoints.Their material distributions are complementary: Cyberpunk 2077 contains more high-metallic pixels, while Black Myth: Wukong contains more high-roughness regions.
- 3.4 Data Collection: The dataset includes annotations for texture, weather, scene type, and camera–scene motion, while static clips and excessively low-luminance frames are filtered out.Five frames are uniformly sampled per clip for annotation, with weather categories including sunny, cloudy, foggy, rainy, and snowy.
- 3.5 Dataset Post-processing: Sharp RGB frames are captured with engine motion blur disabled, then an offline motion-blurred variant is synthesized by averaging eight RIFE-interpolated sub-frames in the linear domain.The paired variants support pixel-aligned tasks while reducing the gap to real videos containing camera-induced blur.
4 VLM-based Evaluation on Real-Scene Test Cases
Real-scene evaluation is difficult because material ground truth is generally unavailable and expert user studies scale poorly. The paper therefore motivates VLMs as scalable judges for relative material comparisons using semantic and appearance priors.
- Evaluation Motivation: Real-capture material predictions lack ground truth, while expert user studies require substantial effort and do not scale well.The evaluation challenge is especially pronounced for material properties in complex scenes.
- VLM-based Evaluation: VLMs can provide scalable relative comparisons because their material-related world knowledge supports pairwise preferences for metallic and roughness.These properties offer semantic categories and characteristic specular behavior that can make comparisons more consistent.
5 Experiments
Experiments evaluate inverse rendering, relighting, and game editing using synthetic benchmarks, real-world videos, user studies, and ablations. Fine-tuning on the proposed dataset improves decomposition, temporal robustness, relighting, and controllable editing.
- 5.1 Training and Experimental Setup: Fine-tuning the official DiffusionRenderer implementation tests whether the proposed dataset improves real-video inverse-rendering generalization.The study also compares against two diffusion-based image inverse-rendering models.
- 5.1 Training and Experimental Setup: The Black Myth: Wukong benchmark holds out 39 clips of 57 frames spanning diverse materials, lighting, and dynamic events.The fine-tuned model achieves the best depth and normal estimation and strongest scale-invariant albedo scores.
- 5.1 Training and Experimental Setup: The fine-tuned model achieves the best overall performance on both depth and albedo on MPI-Sintel’s final pass.Sky-containing samples are excluded from albedo evaluation because Sintel’s annotation differs from the dataset convention.
- 5.2 Quantitative Evaluation of Inverse Rendering: VLM evaluation ranks predictions on temporal consistency, spatial quality, and semantic plausibility for 40 real-world videos.Fine-tuning improves all metrics, while motion augmentation further improves results except for roughness; expert judgments generally align with VLM rankings.
- 5.3 Qualitative Evaluation of Inverse Rendering: Qualitative results show cleaner albedo, more precise depth and normals, and semantically accurate metallic and roughness predictions under smoke and volumetric scattering.The results are compared against DiffusionRenderer on real-world video sequences.
- 5.2 Quantitative Evaluation of Inverse Rendering: Motion augmentation improves most synthetic metrics and increases real-video temporal stability, especially under strong motion blur.It reduces flicker and boundary crawling in the ablation.
- 5.5 Evaluation of Relighting: Improved G-buffers enable an unfine-tuned forward renderer to produce more realistic, illumination-consistent novel views under target environment maps.The advantage is especially visible in sky regions where baseline models often struggle.
- 5.6 Evaluation on Game Editing: The game-editing model balances editability with fidelity, while G-buffers provide stable conditioning for geometry, materials, and complex atmospheric effects.Compared alternatives suffer from temporal instability, excessive input deviation, or limited aggressive-editing capability.
6 Conclusion
The paper presents a large-scale game-derived video dataset and training recipe that unify inverse and forward rendering. Experiments support improved material decomposition, G-buffer-conditioned synthesis, temporal coherence, and controllable editing in the wild.
- 6 Conclusion: The dataset unifies inverse and forward rendering with long-term RGB sequences synchronized with dense G-buffers and motion-blur variants.It is curated from high-fidelity commercial games.
- 6 Conclusion: The VLM-based protocol evaluates in-the-wild performance without ground truth using a scalable assessment framework.The protocol addresses real-world evaluation where intrinsic-buffer ground truth is unavailable.
- 6 Conclusion: Fine-tuning DiffusionRenderer on the dataset substantially improves robust material decomposition and G-buffer-conditioned video synthesis.The conclusion presents this as the principal experimental outcome.
- 6 Conclusion: The data and training recipe support reliable, temporally coherent bidirectional rendering and controllable generative editing in the wild.The conclusion frames these capabilities as a foundation for advanced world simulation.
A.1 License Statement and Data Release Policy
The dataset is collected from two commercial games through a rendering-API pipeline designed to comply with game licenses. Release is non-commercial, gated, and accompanied by an open-source curation toolkit.
- Data Sources: The dataset uses visual and geometric data from Cyberpunk 2077 and Black Myth: Wukong.The paper describes the titles as commercial games and outlines their intellectual-property compliance framework.
- Data Collection Methodology: The curation toolkit intercepts runtime G-buffers and final rendering outputs at the graphics-API level.It does not decompile executables, circumvent anti-tamper mechanisms, or extract proprietary source assets.
- Compliance and Licensing: The dataset is released under CC BY-NC-SA 4.0 for non-commercial research purposes.The policy follows the developers’ Fan Content Policies and EULAs.
- Gated Access: Access is gated through a Terms of Use agreement acknowledging original copyrights and requiring non-commercial use.The paper will not provide direct public download links.
- Open-Source Toolkit: The data curation toolkit will be fully open-sourced to support reproducible collection from other games.The stated goal is continued expansion and diversification of such datasets.
A.2 VLM Evaluation Prompt
The appendix specifies the prompt and fixed video layout used to evaluate metallic predictions with a vision-language model. Three candidate predictions are compared against an RGB reference.
- VLM Evaluation Prompt: The evaluation prompt asks an expert VLM to assess three metallic predictions shown in a 2×2 grid video.Metallic predictions are grayscale or single-channel, with higher values representing greater metallicity.
- VLM Evaluation Prompt: The fixed layout places the RGB reference at top-left and predictions A, B, and C at the other three positions.Prediction A is top-right, B bottom-left, and C bottom-right.