Source-linked AI summary
GaussianRenderFormer: Transformer-based Rendering of 3D Gaussian Splats
Shahaf Valencia Levy, Sagie Benaim
TL;DR
Can a network render 3D Gaussian splats directly, without a rasterizer? GaussianRenderFormer uses a transformer to map Gaussians and a camera to an image; on unseen objects it remains below rasterization of the same pruned input, with diagnostics consistent with a view-decoder bottleneck.
Problem
Rendering 3D Gaussians requires resolving visibility and blending among many overlapping, semi-transparent primitives, while existing Gaussian methods still use a hand-written rasterizer to produce images.
Source: §1 · p. 2Method
GaussianRenderFormer adapts a two-stage transformer to render Gaussian tokens directly from a set of splats and a camera, without a rasterizer in the renderer.
Source: §4 · p. 4Results
38.08 dB PSNR on 300 unseen objects versus 44.02 dB for rasterizing the same pruned input, measured against renders of the full, unpruned splats.
Source: §1 · p. 2Takeaways & Limitations
A learned renderer for Gaussians is feasible, but the reported model is not yet competitive with a rasterizer.
Source: §8 · p. 12The model is not yet a practical rasterizer replacement: it remains several dB lower in quality, is several hundred times slower, and needs far more memory.
Source: §7 · p. 12
Abstract
from arXiv · showhide
Can a neural network learn to render 3D Gaussian splats on its own, with no rasterizer in the renderer? We present GaussianRenderFormer, a transformer that takes a set of Gaussians and a camera and outputs the image directly. Each Gaussian becomes one token in a view-independent encoder, and a view-dependent decoder turns small patches of camera rays into pixels by attending to those tokens. Trained on 26,820 Objaverse objects pruned to 20,000 Gaussians each, GaussianRenderFormer renders 300 unseen objects at 38.08 dB PSNR against the full, unpruned splats, while rasterizing the same pruned input reaches 44.02 dB. The gap is not one of generalization, and our diagnostics are consistent with a bottleneck in the view decoder: telling it the projected footprint of each Gaussian, which it could in principle infer, adds 1.4 dB. Under-training is not ruled out. Code, trained models and videos: https://svlwoof.github.io/gaussianrenderformer
1 Introduction
GaussianRenderFormer replaces the hand-written 3D Gaussian Splatting rasterizer with a neural renderer, addressing the challenge of resolving visibility and blending among many overlapping, semi-transparent Gaussians. The paper presents this as a feasibility study: it renders unseen objects, but remains below rasterization and its expected broader benefits are not demonstrated.
1 Introduction
- GaussianRenderFormer replaces the rasterizer with a network that takes Gaussians and a camera and outputs an image directly, without per-scene optimization.Its contributions include adapting a mesh renderer’s view decoder with image-plane position and footprint information, smaller patches, and a schedule that trains the new input layer first.Source: §1 · pp. 1–2
- A learned renderer could learn light transport, combine splats with meshes and lights, avoid sorting-related popping, or learn a prior for missing input, but this paper demonstrates none of those benefits.The authors state that a network must first render Gaussians at all.Source: §1 · p. 2
- Rendering requires resolving visibility and blending at each pixel among heavily overlapping, semi-transparent Gaussians, unlike the surface-based triangles RenderFormer handles.Objects typically contain tens of thousands of Gaussians, while RenderFormer was trained on scenes of up to 4,096 triangles.Source: §1 · p. 2
- 38.08 dB PSNR on 300 unseen objects is below 44.02 dB from rasterizing the same pruned input, against renders of the full, unpruned splats.The authors describe this as an object-level feasibility study; diagnostics are consistent with a view-decoder bottleneck, though under-training is not ruled out.Source: §1 · p. 2
2 Related Work
GaussianRenderFormer differs from prior neural rendering approaches by taking explicit Gaussians as input and producing images with a network rather than a hand-written renderer. Related work either learns to render other scene representations or uses Gaussians in pipelines that still rasterize the output.
2 Related Work
- Prior Gaussian-rendering algorithms project and blend splats, ray-trace them, or replace them with ray-traceable ellipsoids; these remain hand-written algorithms.Sort-free Gaussian Splatting learns replacement sorting weights but retains rasterizer projection and blending.Source: §2 · pp. 2–3
- GaussianRenderFormer takes explicit Gaussians as input and renders the image with a network, whereas prior Gaussian reconstruction, analysis, and relighting methods still use a rasterizer wherever they produce an image.Learned renderers for meshes, points, voxels, and latent scene codes provide related approaches but do not take Gaussians as input.Source: §2 · p. 3
- Ray-query view-synthesis methods such as LVSM use images rather than explicit Gaussians as scene input, so their outputs cannot be compared directly with rasterizing the same input.GaussianRenderFormer’s explicit input permits that direct comparison.Source: §2 · p. 3
3 Background
The background describes the rasterizer computation GaussianRenderFormer replaces and the two-stage transformer layout it adapts from RenderFormer. The adaptation changes the scene tokens and how the view-dependent decoder attends to the scene.
3.1 3D Gaussian Splatting
- 3DGS represents a scene as Gaussians with centers, covariances, opacity, and color; the rasterizer projects them to 2D, sorts them by depth, and blends them front to back at each pixel.GaussianRenderFormer replaces this computation and uses only view-independent RGB color in its input and ground truth.Source: §3.1 · p. 3
3.2 RenderFormer
- RenderFormer uses a view-independent stage of triangle tokens and self-attention, followed by a view-dependent stage where ray-patch tokens cross-attend to scene tokens before a head produces pixels.GaussianRenderFormer keeps this two-stage layout and pretrained weights, but uses one token per Gaussian and modifies scene attention for the many semi-transparent Gaussians.Source: §3.2 · p. 4
4 Method
GaussianRenderFormer retains RenderFormer’s two-stage layout but adapts it to Gaussian tokens and a view-dependent patch decoder. Camera-ray patch tokens attend to scene tokens, and the decoder produces pixels through a DPT head.
4.1 Gaussian tokens and the scene encoder
- Each Gaussian’s 14 values are linearly embedded and RMS-normalized into a token, while 12 unchanged self-attention layers exchange information using 3D rotary positions of Gaussian centers.Source: §4.1 · pp. 4–5
4.2 The view decoder
- Six layers of cross-attention and patch self-attention map camera-ray tokens to pixels through a DPT head, leaving visibility, blending, and color to the network rather than per-pixel computation.Source: §4.2 · pp. 5–6
- Projected image positions enter attention through 2D rotary embeddings, making patch–Gaussian attention depend on their distance in the image.Source: §4.2 · p. 5
- Using 4 × 4-pixel patches gives the decoder four times as many tokens as 8 × 8 patches, while projected Gaussian footprints alter cross-attention keys rather than values.Source: §4.2 · p. 5
4.3 Training
- Training targets full-splat renders from 20,000-Gaussian inputs using log-encoded L1 and VGG-LPIPS losses, object-weighted pixels, and shared random rotations of each object and camera.Source: §4.3 · p. 6
- Training proceeds from a new-input-layer stage to whole-model training, then short recovery with the decoder changes and final 512 × 512 training with LPIPS.Source: §4.3 · p. 6
5 Data
The data pipeline selects well-fitting Objaverse splats, normalizes them, and prepares pruned inputs paired with full-splat targets. Separate development and fresh held-out object sets support evaluation.
- Filtering ShapeSplat’s Objaverse splats for fit and opacity leaves 26,820 training objects and 1,806 validation objects, each with exactly 50,000 Gaussians.Source: §5 · p. 6
- Each splat is centered on its median Gaussian position, oriented with up along +y, and uniformly scaled into [−0.45, 0.45]³; only view-independent Gaussian colors are retained.Source: §5 · p. 6
- Full-splat targets comprise 28 shared orbit-camera views per object at 512 × 512 pixels, with a 45° field of view and three camera distances.Source: §5 · p. 6
- Inputs retain the 20,000 highest-scoring Gaussians from each 50,000-Gaussian splat, then receive 1,500 fine-tuning steps against full-splat renders from 64 views.Source: §5 · p. 7
- The development held-out set contains 300 objects used for design decisions, while the fresh held-out set uses 300 additional objects not used during development.Source: §5 · p. 7
6 Experiments
On 300 fresh held-out objects, GaussianRenderFormer renders the full-splat reference less accurately than rasterizing its pruned input. Ablations and diagnostics examine decoder design, training, and the configurations where the model works.
6.1 Main result
- 38.08 dB PSNR against the unpruned render, versus 44.02 dB for rasterizing the same pruned input, on 300 fresh held-out objects.Across the three training distances, the gap ranges from 3.9 dB to 6.8 dB; model errors concentrate on edges and fine texture.Source: §6.1 · p. 7, Table 2 · p. 8
6.2 Ablations
- The harness ablations use 1,000 objects and single runs, but the all-changes recipe varies by only 0.02 dB across two runs on held-out objects.The all-changes arm must learn all three decoder adaptations during the harness steps, which favors the ablated arms.Source: §6.2 · pp. 8–9
6.1 Main result
- At 20,000 Gaussians, one view takes 275 ms on an L40S GPU, compared with under a millisecond for gsplat.The scene encoder accounts for 151 ms of the 275 ms; time grows roughly quadratically with input size, reaching 75 s at 640,000 Gaussians.Source: §6.1 · pp. 7–8, Table 3 · p. 9
6.2 Ablations
- Without the projected footprint, the final model loses 1.4 dB on fresh held-out objects; removing the 2D embedding or using 8 × 8 patches also lowers harness performance.On the harness, those changes cost 5.2 dB and 3.0 dB, respectively; the improvements also appear on training objects.Source: §6.2 · p. 9, §6.4 · p. 11
6.3 Data and training length
- Fine-tuning on more objects closes the training-versus-rasterizer gap, but even the final model trained about 10 times longer scores similarly on training and held-out objects.The final run was still gaining about 0.9 dB with each doubling of steps from 94,000 to 375,000, so under-training remains possible.Source: §6.3 · p. 10
6.4 Where the gap comes from
- Diagnostics are consistent with a view-decoder bottleneck: explicitly supplying each Gaussian’s projected footprint adds 1.4 dB, though the decoder could infer it from its inputs.The model also blurs or adds fine detail, and denser inputs hurt its score while helping rasterization.Source: §6.4 · pp. 10–11
6.5 Beyond the training setup
- The model is tied to its training camera configuration: changing resolution or distance can substantially reduce quality, and 40,000-Gaussian inputs lower its score by 3.1 dB.Changing field of view costs 1.1 dB at 35° and 1.2 dB at 60°; varying configurations during training is expected to help but requires more training.Source: §6.5 · pp. 11–12
- On smooth held-out camera orbits, the model averages 38.52 dB per frame without flicker, despite never being trained on sequences.On four real-scene regions viewed from outside on an orbit, it gets shapes, colors, and layout right but softens fine texture; these are not full-scene renders.Source: §6.5 · p. 12
7 Limitations
The renderer remains several dB below rasterization of the same input, is much slower and more memory-intensive, and is limited in scene scope, camera configuration, appearance, and input preparation. Its design also depends on RenderFormer’s pretrained weights, and training is expensive.
Quality and speed
- GaussianRenderFormer remains several dB below rasterizing the same input, is several hundred times slower, needs far more memory, and scales poorly with input size.The authors therefore do not consider it a practical rasterizer replacement yet.Source: §7 · p. 12
Objects, not scenes
- The study covers single objects viewed from outside; full-scene rendering, including views from inside a scene, remains open.Full self-attention in the scene encoder also limits the number of Gaussians the model can take.Source: §7 · p. 12
One camera configuration
- GaussianRenderFormer renders well only in its training camera configuration; changes in input size, distance, resolution, or field of view reduce quality.Unlike the model, the rasterizer does not incur these quality costs in the reported comparisons.Source: §7 · p. 12
Appearance
- Using only view-independent Gaussian colors prevents rendering spherical-harmonic view-dependent effects, and fixed-lighting colors cannot be relit.Source: §7 · p. 12
Input preparation
- Preparing inputs from larger splats requires rasterizing each Gaussian from 64 views for pruning, followed by ordinary 3DGS optimization; inputs up to 20,000 Gaussians need neither step.Source: §7 · p. 12
Bound to RenderFormer’s design
- The model depends on RenderFormer’s pretrained weights and expensive training, and the authors found no recipe that trains it from random weights.Changing pretrained parts could undo what their weights encode, while adding capacity in an earlier version gave no improvement.Source: §7 · p. 12
8 Conclusion
Along smooth camera paths, the model’s per-frame quality matches its evaluation-view quality, though its consistency score trails gsplat’s. The paper concludes that rasterizer-free Gaussian rendering is feasible but not yet competitive, and identifies materials and lights as future directions for relighting and global illumination.
8 Conclusion
- GaussianRenderFormer renders unseen objects without a rasterizer in the renderer, but a clear gap to rasterizing the same input remains.The authors’ diagnostics are consistent with a view-decoder bottleneck, although under-training is not ruled out.Source: §8 · p. 12
- Adding Gaussian material parameters and scene light sources could let a model like this learn relighting and global illumination.Source: §8 · p. 13
A Camera paths
- The model’s camera-path consistency score is 4.8 dB below gsplat’s, while its per-frame gap is 5.6 dB; the videos show no visible flicker.The path evaluation covers 30 held-out objects over 96 frames, with distance varying within the training range.Source: A Camera paths · p. 17, Table 5 · p. 17, Figure 5 · p. 17
B Real captures
On four Mip-NeRF 360 real captures, GaussianRenderFormer preserves scene shape, color, and layout at its 20,000-Gaussian input size, but trails rasterization and softens fine texture. Supporting measurements cover full-scale ablations, departures from the training setup, training progress, and variation across held-out objects.
B Real captures
- At 20,000 Gaussians, GaussianRenderFormer trails rasterizing the same input by 1.6–4.7 dB on four real captures, while preserving shape, color, and layout but softening fine texture.The scenes are bonsai, counter, garden, and kitchen; the reported means cover seven views. Boundary-cut and long thin edge Gaussians create blurry streaks in both the reference and model, which the model smears further.Source: B Real captures · pp. 17–18, Table 6 · p. 18, Figure 6 · p. 19
C.1 Supporting measurements
- With the footprint, the full-scale model reproduces 85% of the reference’s content above 1/4 cycles per pixel, versus 83% without it.The added content absent from the reference falls by 24–38% between 1/16 and 1/4 cycles per pixel; the supporting measurements also report 0.19 dB more held-out PSNR for a harness trained on its own-input target than one trained on the full-splat target. A fine-tune on one object reached 45.7 dB with only the decoder trained and 46.4 dB with the whole model.Source: §C.1 · p. 21, Table 7 · p. 21
C.2 Beyond the training setup
- Table 8 tests one-factor changes from the 20,000-Gaussian, distance-1.7, 512 × 512, 45° training setup on 30 held-out objects.It reports PSNR against the full-splat rasterization; the supplied passages do not give its cell values.Source: §C.2 · p. 21, Table 8 · p. 22
C.3 Cost and input size
- The input-size cost measurements cover one 512 × 512 view from 20,000 to 640,000 Gaussians on one NVIDIA L40S GPU.The table reports time and peak memory; the model’s memory includes its weights, while gsplat’s excludes the input.Source: §C.3 · p. 21, Table 9 · p. 22
C.5 Training curve
- The final run was still improving at its end, though by less in each successive quarter.Held-out PSNR is reported at four checkpoints after 10,000-step learning-rate decays, and Figure 8 shows the validation loss at 512 pixels.Source: §C.5 · p. 22, Figure 8 · p. 22, Table 11 · p. 23
C.4 First training stages
- Table 10 compares first-stage training recipes, each run for 50,000 steps at 256 × 256 pixels on all 26,820 objects.It reports held-out PSNR and LPIPS against the unpruned render, differences from the authors’ recipe, and final validation loss; the supplied passages give no cell values.Source: §C.4 · p. 21, Table 10 · p. 23
C.6 Renders of the ablations
- In the rendered ablations, the full model keeps a painted pattern that the footprint-free version renders slightly softer.The shorter-trained harness models are softer and lose much of the pattern without the footprint, without the 2D embedding, and with 8 × 8 patches.Source: Figure 9 · p. 23, §C.6 · p. 23
C.7 Spread over objects
- Across 300 fresh held-out objects, GaussianRenderFormer’s PSNR ranges from 26.5 to 45.0 dB, with a median of 38.2 dB.Gsplat on the input ranges from 36.4 to 49.3 dB.Source: §C.7 · p. 23
C.8 Error on edges
The diagnostics locate GaussianRenderFormer’s largest errors in object edges and fine detail, while overlap and failure-case analyses further characterize where rendering is difficult. The development history also records why training shifted to single Objaverse_Splats objects.
C.8 Error on edges
- Rasterizing the input places 78% of its error on these pixels; the model trails the rasterizer by 6.2 dB on edges and 8.1 dB on the rest of the object.Source: §C.8 · p. 24
C.9 Error by frequency band
- At the highest frequencies, the model reproduces 85% of the reference content versus 93% for the rasterizer, while adding almost four times as much non-reference content.Its excess error relative to the rasterizer is largest at the lowest frequencies, and fitting per-view, per-channel gain and bias raises its PSNR by only 0.05 dB.Source: §C.9 · p. 24, Table 12 · p. 24
C.10 Error by overlap
- About 130 of the 20,000 input Gaussians reach a typical object pixel, and 77% of pixels are reached by at least 32.Both renderers score best on single-Gaussian pixels and worst on pixels reached by 16–63 Gaussians, mostly on silhouettes and thin parts; their gap is smallest at one to three Gaussians.Source: §C.10 · p. 25, Table 13 · p. 25
C.11 Failure cases
- On four lowest-scoring fresh held-out objects at their worst views, GaussianRenderFormer scores 25.4, 27.2, 29.1, and 28.5 dB, versus 37.9, 40.0, 38.6, and 40.4 dB for gsplat on the input.The shapes are right; errors include dense textures rendered less precisely and soft, low-contrast structure.Source: Figure 11 · p. 26, §C.11 · p. 26
D The road to the final model
- The authors summarize earlier experiments because several final design choices followed from approaches that did not work.Most early results used other protocols and are reported qualitatively.Source: D The road to the final model · p. 26
D.1 Data
- The initial training set used meshes sampled into Gaussians and RenderFormer renders as targets, but the model learned to imitate the lighting rather than render its input.The authors switched to real splats rendered by gsplat; tests on outside captures also suggested input quality mattered more than input size.Source: §D.1 · pp. 26–27
D.2 Training schedule
Training the input layer first, larger inputs, augmentation, and longer training improved the model; subsequent decoder changes further improved training and held-out results. Earlier trials of other changes did not justify full-scale runs.
D.2 Training schedule
- Training the input layer alone first fixed runs that otherwise stalled at the same loss, and larger inputs, random rotations, and longer training each helped clearly.The best model in this phase, trained on 13,405 objects, scored 30.39 dB on the held-out set.Source: §D.2 · p. 27
D.3 The decoder
- The 2D rotary embedding of projected positions was the first decoder change to improve both training and held-out objects; smaller patches improved them further.Using bf16 with gradient checkpointing made a training step about three times faster, enabling smaller patches; the final model combined these changes and trained on 26,820 objects for 375,000 steps.Source: §D.3 · p. 27
E Changes that did not help
- None of the tested changes that did not help justified a full run, though most were tested on small training sets with earlier model versions.The table reports training and held-out margins as rasterized-input PSNR minus model PSNR, so lower margins are better.Source: E Changes that did not help · p. 27, Table 14 · p. 28
F Training on a single object
Single-object training separates the network’s ability to represent detail from its ability to generalize across objects. It can restore detail removed by pruning, but the authors do not consider this approach practical.
F Training on a single object
- The single-object experiments use earlier model versions and test what the architecture can represent separately from what shared weights can generalize.The authors do not treat per-object training as a practical use of the model.Source: F Training on a single object · p. 28, §F.3 · p. 30
F.1 How much detail can the network output
- On 14 memorized tomato-scan training views, the model reached 52.5–53.0 dB against the full splat and its renders could not be distinguished from the reference.On new views between training views, it was sharper than rasterizing the input and scored 30.5 versus 29.7 dB in PSNR and 0.09 versus 0.15 in LPIPS against the full splat.Source: §F.1 · pp. 28–29, Figure 12 · p. 29
F.2 Compensating for pruning on one object
- With 9,000 views of one object, per-object training beat rasterizing the pruned input on 4 of 13 tested objects, by up to 1.8 dB.The gain came from learning the object and filling in pruned detail, not from rendering the input more faithfully.Source: F Training on a single object · p. 28, §F.2 · p. 29, Table 15 · p. 30
F.3 Why we do not pursue it
- Per-object training is not considered practical: each object needs its own 195M-parameter network (about 745 MB), and rendering took 476 ms versus 1 ms for gsplat.The gain was under 2 dB where it worked; training also took four to five cycles of 30,000 steps per object, and LoRA fell short of the rasterizer at equal storage.Source: §F.3 · pp. 30–31
G Reproducibility
The paper releases code, trained weights, experiment scripts, and evaluation-object lists, with a documented route to reproduce later training stages. Code and trained weights are released under MIT; the Objaverse objects retain their creators’ licenses.
G Reproducibility
- The released code, weights, experiment scripts, and object lists include commands and expected numbers for every table and figure.Starting from the released stage-two checkpoint reproduces stages three and four; the full four-stage run is documented but has not been run end to end.Source: G Reproducibility · p. 31
- Stages one through four used 46, 662, 592, and 27 GPU-hours, respectively.The first two stages used NVIDIA L40S GPUs; stages three and four used A40 and RTX Pro 6000 GPUs, and the final decay used L40S GPUs.Source: G Reproducibility · p. 31
Licenses
- The code and trained weights are released under MIT, while Objaverse_Splats has no separate license and its objects retain their creators’ Creative Commons licenses.The underlying Objaverse collection is released under ODC-By 1.0.Source: Licenses · p. 32
Related papers
This paper cites
- The Unreasonable Effectiveness of Deep Features as a Perceptual Metric2018
- RoFormer: Enhanced Transformer with Rotary Position Embedding2021
- Root Mean Square Layer Normalization2019
- Objaverse: A Universe of Annotated 3D Objects2022
- 2D Gaussian Splatting for Geometrically Accurate Radiance Fields2024
- Mip-Splatting: Alias-free 3D Gaussian Splatting2023