Source-linked AI summary
RenderFormer-V2: Neural Rendering with Heterogeneous Scene Primitives
Chong Zeng, Yue Dong, Pieter Peers, Lvmin Zhang, Maneesh Agrawala
TL;DR
RenderFormer-V2 addresses the limited scalability and versatility of prior neural renderers, especially RenderFormer’s handling of large meshes, materials, and lighting. It uses a two-stage transformer with rendering-aware sparse attention, heterogeneous primitives, and latent material appearance embeddings. The model renders varied complex scenes without per-scene training or specialized code, while remaining limited by eight light sources, single-frame training, and fixed 32 × 32 texture resolution per triangle.
Problem
Prior neural renderers have limited generalizability or require per-scene training, while RenderFormer is constrained by quadratic attention, hard-coded materials, and limited scene primitives and lighting.
Method
RenderFormer-V2 models global light transport with a two-stage transformer using windowed and render-aware sparse attention, heterogeneous primitives, and latent material appearance embeddings.
Results
RenderFormer-V2 renders more complex and larger scenes with varied lighting and materials, including refractive surfaces, caustics, environment lighting, volumetric scattering, textures, displacement, and many primitives.
Takeaways & Limitations
The learned renderer supports full global illumination across heterogeneous scene inputs without per-scene training or specialized code for displacement mapping and volumetric scattering.
Takeaways & Limitations
The model remains limited to eight light sources per scene, is trained on single frames without explicit temporal coherence, and uses fixed 32 × 32 texture resolution per triangle.
Abstract
from arXiv · showhide
We present 'RenderFormer-V2', a unified learned transformer-based neural rendering model, complementary to modern physics-based rendering systems, that can handle diverse light-transport effects such as caustics, volumetric scattering, environment lighting, textured and displaced surfaces and out-of-distribution materials without per-scene training or specialized code. RenderFormer-V2 models global light transport as a sequence-to-sequence transformation. Following its predecessor, RenderFormer-V2 also employs a two stage process: a view-independent stage that resolves intra-scene primitive to primitive transport, and a view-dependent stage that transforms the internal neural scene representation into image pixels. Different from RenderFormer, our model employs a novel combined windowed-attention and rendering-informed attention sink in the view-independent stage to improve scalability while maintaining render accuracy. To further improve versatility, RenderFormerV2 supports heterogeneous scene primitives, including environment maps and participating media, and it employs a material encoding independent of the underlying surface reflectance model that encodes material appearance via a novel neural embedding. We demonstrate the versatility of RenderFormer-V2 on a variety of scenes and perform an extensive ablation of the improved attention mechanism.
1 Introduction
RenderFormer-V2 extends transformer-based neural rendering toward larger, more varied scenes by addressing RenderFormer’s scaling, primitive, material, and lighting limitations. It combines sparse attention, latent material embeddings, and heterogeneous lighting primitives, and demonstrates broader scene coverage with ablations.
- The system targets neural rendering that learns relations among geometry, materials, and light instead of relying on manually encoded light-transport rules.Prior solutions are described as having limited generalizability or requiring per-scene training.
- RenderFormer is limited to fewer than 4k triangles, hard-coded GGX BRDF materials, and at most eight triangular diffuse light sources.
- RenderFormer-V2 combines windowed attention with render-aware attention sinks to reduce view-independent attention bottlenecks for larger primitive sequences.The design addresses quadratic attention cost and attention focus loss in large meshes.
- A latent material embedding represents different BRDF models, including transparent and measured materials, without tying the representation to a hard-coded BRDF.A pretrained VAE encodes 32 × 32 texture patches for spatially varying materials.
- Heterogeneous primitives encode lighting types ranging from triangular light sources to environment maps through specialized tokens.
- RenderFormer-V2 renders more complex and larger scenes with greater lighting and material variety and evaluates its architectural choices through an in-depth ablation study.The paper reports a trained model and code release.
2 Related Work
Prior neural rendering methods trade generalization, scene coverage, or computational scalability, motivating attention-based models that operate over 3D primitives. Transformer scaling methods address long contexts through local, compressed, hierarchical, or sink-based attention, but light transport requires both local and nonlocal interactions.
- Early neural rendering methods often overfit to one or a limited number of scenes, while image-space methods use G-buffers and produce same-view shaded images.
- RenderFormer models transport between triangles and from triangles to the camera, but its brute-force attention does not scale well to large meshes.
- Classic transformer attention has quadratic complexity in sequence length and can lose focus as attention spreads across more tokens.
- Windowed attention reduces computation by restricting attention to nearby tokens, but it can ignore long-range interactions important for light transport.
- Static sparse attention is highly dependent on matching the sparsity pattern, whereas scene light transport does not follow a predetermined pattern.
- Native-Sparse Attention combines sliding-window, compressed, and fine-grained streams, but its secondary retrieval stage raises computational cost.
- Hierarchical attention uses multi-resolution tokens selected by distance, implicitly assuming positional distance correlates with token importance.
- Attention-sink methods reserve dedicated tokens for global relations alongside local sliding-window attention to prevent attention from being dumped into arbitrary tokens.
3 Background - RenderFormer
RenderFormer is an end-to-end transformer neural renderer that converts scene triangles and camera parameters into globally illuminated images. Its two stages first model triangle-to-triangle transport and then transform view-conditioned transport into pixel values.
- RenderFormer takes triangle sequences with GGX BRDF parameters, emittance strength, and camera parameters as input and outputs a rendered image with full global illumination.
- View-independent stage: The view-independent stage uses self-attention to transform embedded triangle tokens into per-triangle tokens encoding triangle-to-triangle light transport.
- View-dependent stage: The view-dependent stage applies cross-attention and self-attention to view-bundle tokens representing 8 × 8 camera-ray grids.
- View-dependent stage: A dense vision transformer converts transformed ray-bundle tokens into pixel values for each ray.
4 Overview
RenderFormer-V2 extends the two-stage transformer rendering pipeline to heterogeneous scene primitives and light sources, while improving scalability and efficiency for larger scenes.
- RenderFormer-V2 retains a two-stage pipeline: view-independent intra-primitive transport followed by view-dependent ray-bundle transformation into output tokens.
- Unlike RenderFormer, RenderFormer-V2 supports heterogeneous scene primitives and different types of light sources.
- The revised system scales better in efficiency and accuracy to larger numbers of scene primitives.
5 Scene Embedding
RenderFormer-V2 represents scenes as heterogeneous tokens with uniform positional encoding and material embeddings designed around appearance rather than a fixed reflectance model.
- Scenes are represented as heterogeneous tokens encoding geometry, material, lighting, and camera information with a uniform relative positional encoding strategy.
- Triangle & Material Embedding: Triangle tokens combine vertex, normal, and material embeddings into a single scene representation.
- Triangle & Material Embedding: The material latent space can encode measured BRDFs outside the training set.
- Triangle & Material Embedding: The learned material appearance embedding is independent of a particular BRDF model and lets the transformer map appearance directly to pixel values.
- Textures, displacement maps, participating media, triangular lights, and environment maps are encoded through specialized token representations.
- Environment maps use separate LDR and HDR token representations augmented with lighting-direction and strength information.
6 RenderFormer-V2 Architecture
RenderFormer-V2 modifies attention in both stages, combining resolution-independent windowed attention with rendering-informed global attention to address scaling and fidelity challenges.
- View-dependent Stage: The view-dependent stage uses world coordinates and shifted SWIN windowed attention instead of full self-attention.The window size is 8 with a shift of 4.
- View-dependent Stage: Windowed attention changes the ray-related cost from O(T × R + R^2) to O(T × R + W^2) and makes the context window resolution independent.
- View-independent Stage: The view-independent stage combines local primitive attention with global-token attention because global illumination and large-scale occlusion affect transport beyond local neighborhoods.
- View-independent Stage: Hilbert-curve sorting linearizes geometry tokens so sliding-window attention can exploit spatial locality efficiently.The sliding window includes 256 tokens before and after the target token.
- View-independent Stage: Attention sinks always include rendering-relevant global tokens in attention calculations.
7 Training
Training uses procedurally generated scenes and a staged curriculum that progressively adds material, lighting, volumetric, scene-complexity, and resolution challenges.
- Training scenes contain randomly selected objects placed in template environments and use randomly generated materials from several homogeneous and SVBRDF categories.
- The model uses the RenderFormer loss: logarithmic-image L1 error combined with a weighted LPIPS perceptual loss.The objective is L_1(log I) + 0.05 LPIPS(clamp(log I / log 2, 0, 1)).
- A five-stage curriculum first learns coarse light transport, then adds materials, environment lighting, volumetric effects, attention changes, larger primitive budgets, and higher resolutions.
- Training reaches a 64k primitive budget and 2048 × 2048 resolution over 19 days on 32× A100 GPUs.
- After pretraining, rendering new scenes requires no fine-tuning or additional training.
8 Results
RenderFormer-V2 renders larger and more diverse scenes, including complex materials, lighting, volumetric effects, and fine geometry. Its sparse attention improves scalability and quality, while ablations support the importance of the complete attention design.
- RenderFormer-V2 renders refractive surfaces with caustics, environment lighting, volumetric scattering, textures, displacement mapping, and large primitive counts without specialized code.
- A latent material appearance space supports materials outside the training set, including selected measured materials from the RGL BRDF dataset.
- RenderFormer-V2 achieves better accuracy than RenderFormer on scenes with many triangles, avoiding the attentional-loss darkening observed at 83.2k triangles.
- Higher-resolution rendering improves fine geometric detail, including dragon claws and Lucy’s face and hands, at 2048×2048 compared with 512×512.
- All components of the sparse attention mechanism are necessary to achieve the highest render quality in the ablation study.
- The method retains practical constraints, including a maximum of eight light sources, fixed 32×32 texture resolution per triangle, and substantial retraining for new primitives.
9 Conclusion
RenderFormer-V2 is a transformer-based neural renderer that maps heterogeneous scene primitives to images with full global illumination. Its rendering-aware sparse attention improves scaling to larger scenes, while end-to-end learning supports effects without specialized handling.
- RenderFormer-V2 maps sequences of heterogeneous primitives, including textured triangles, volumetric elements, light sources, environment maps, and cameras, to rendered images with full global illumination.
- Rendering-aware sparse attention resolves intra-primitive light transport while improving scaling for larger scenes during training and inference.
- The flexible latent material appearance space specifies surface reflectance, while end-to-end training avoids specialized code for displacement mapping and volumetric scattering.
- Comparisons with path-traced renderings report PSNR, SSIM, LPIPS, and FLIP errors across scenes containing refractive, environment-lit, volumetric, textured, and displaced effects.
C.1 Architecture
RenderFormer-V2 changes both token representations and attention patterns relative to RenderFormer. Heterogeneous tokens, rendering-aware sparse attention, and SWIN attention target broader support, accuracy, and scalability.
- RenderFormer-V2 supports heterogeneous tokens for triangles, volume elements, and environment maps, plus spatially varying materials, displacement mapping, and BRDF-agnostic material specification.
- The view-independent stage replaces dense self-attention with sliding-window attention and rendering-aware sinks containing global registers, light sources, and summarization tokens.
- The view-dependent stage replaces full ray-bundle self-attention with SWIN attention because distant image pixels are unlikely to be meaningfully related.
- Table 2 compares RenderFormer and RenderFormer-V2 at 2048×2048 resolution with and without high-resolution fine-tuning.
- RenderFormer-V2 more accurately renders fine geometric details than RenderFormer in qualitative comparisons against path-traced references.
D Additional Results
RenderFormer-V2 handles prior RenderFormer test scenes, additional complex rooms, and procedurally generated scenes with diverse geometry, textures, lighting, and volumetric effects. The results also expose a texture-resolution limitation for large triangles that subdivision can mitigate.
- RenderFormer-V2 handles all 8 RenderFormer test scenes without issues.
- Additional room scenes contain fine geometrical details, complex light transport, and textures.
- The supplementary video demonstrates additional results and stability under changes in scene and camera parameters.
- Procedurally generated ablation scenes vary textures, test objects, scene token counts, volumetric effects, area lighting, and environment lighting.
- Fixed 32 × 32 texture embeddings reduce effective texture resolution on large triangles, while mesh subdivision can significantly improve texture quality.The limitation is illustrated in Figure 9.