Pulsar: Efficient Sphere-based Neural Rendering

1. Introduction

• End-to-end differentiable rendering requires scalable gradients through scene representation, projection, and shading for complex scenes at real-world resolutions.Source: §1 · pp. 1–2
• Pulsar combines a sphere-based scene representation, efficient differentiable rendering, and neural shading in a modular PyTorch-integrated system.Spheres carry position, radius, opacity, and arbitrary payloads, while volumetric compositing forms images or feature maps.Source: §1 · p. 2
• Up to one million spheres can be rendered and optimized in real time at 1024×1024 resolution on consumer graphics hardware.C++ execution takes less than 22 ms for rendering and less than 6 ms for gradient calculation.Source: §1 · p. 2
• Pulsar executes orders of magnitude faster than existing techniques and enables real-time rendering and optimization of representations with millions of spheres.The design avoids mesh topology changes, sparse representations cull empty space, and sphere rendering needs no acceleration structure.Source: §1 · p. 2
• Pulsar is demonstrated for 3D reconstruction, realistic human-head modeling, and novel-view synthesis.Source: §1 · p. 2

2. Related Work

• Differentiable rendering methods include mesh, voxel, function-based, and point-based scene representations, with neural rendering also implemented solely through deep models.Source: §2 · pp. 2–3
• Pulsar avoids mesh-based geometry while using 3D spheres without normals; extra channels can capture and optimize normal information.Table 1 identifies DSS and PyTorch3D as the only other listed renderers that also avoid mesh-based geometry.Source: Table 1 · p. 3
• Physics-based differentiable renderers model more physical detail but execute more slowly, whereas Pulsar prioritizes fast real-time rendering for neural rendering.The paper states that Pulsar’s optimization strategies would not be implementable in the same way using Difftaichi or Enoki.Source: §2 · p. 3
• Earlier point-based methods were limited to 128 × 128 images and 16k points, or reported slower runtimes such as 258 ms forward and 680 ms backward at 256 × 256 resolution.Source: §2 · p. 3
• Pulsar differs from related point renderers by optimizing sphere radius, using a different blending function, and allowing unlimited sphere contributions to pixel color.Only the first few spheres are used for gradient propagation to avoid high-frequency artifacts found in complex scenes.Source: §2 · p. 3

3.1. Sphere-based Scene Representation

• Pulsar represents scenes as spheres with learned positions, features, radii, and opacities, all optimized through differentiable rendering.The feature vector can encode color, radiance, or abstract features for a neural network.Source: §3.1 · p. 4

3.2. Efficient Differentiable Rendering

• The renderer maps a sphere-based scene and camera parameters to a differentiable feature image, with gradients available for camera rotation, translation, and most intrinsic parameters.This mapping explicitly incorporates the camera rotation R, translation t, and intrinsic parameters K.Source: §3.2 · p. 4
• Pulsar integrates opacity into both depth and distance factors because using only one factor can leave foreground spheres visible or let depth override opacity.The two-factor treatment is described as feasible and numerically stable.Source: §3.2 · p. 4
• Pulsar accelerates rendering with pixel-based parallelization, tile-based candidate filtering, and cooperative filtering across hardware levels.Pixel parallelization avoids synchronization when multiple threads write to the same pixel and makes speed largely independent of sphere size.Source: §3.2 · pp. 4–5
• Early stopping terminates sphere-intersection searches once farther spheres cannot materially affect blending, improving asymptotic scaling.Spheres are sorted by earliest possible intersection, and aggregated weights provide an upper bound for deciding when to stop.Source: §3.2 · p. 5, Figure 3 · p. 5

3.3. Neural Shading

• A neural shading network converts the rendered feature map into RGB output, using either a convolutional U-Net or a per-pixel network with an overfitting-capacity trade-off.Higher shading-network capacity can overfit, so the choice depends on the application and available training data.Source: Figure 2 · p. 4, §3.3 · p. 5

3.4. End-to-end Optimization

• Pulsar jointly optimizes the sphere scene and neural shading model from 2D image observations, while other pipeline stages remain modular PyTorch components.The optimization uses gradient-based methods and does not require additional 3D supervision.Source: §3.4 · p. 5

Runtime Performance

• Pulsar outperforms current state-of-the-art differentiable renderers, including PyTorch3D, by at least a factor of five and sometimes more than two orders of magnitude.The comparison covers runtime on scenes with varying complexity.Source: §4 · p. 5
• At 4K resolution with 4.4M spheres, Pulsar measures 387ms on an NVIDIA RTX 2070 GPU with 3500 MB memory usage.The reported runtime remains below 400ms for this configuration, depending on GPU and scene.Source: §4 · p. 5
• Pulsar scales favorably with sphere count relative to PyTorch3D: PyTorch3D reaches 400 ms below 500K spheres, while Pulsar remains below 400 ms at approximately 4.4M spheres.Pulsar also measured less than 400 ms at 4K resolution with 4.4M spheres, depending on GPU and scene, including 387 ms on an NVIDIA RTX 2070 GPU.Source: §4 · p. 5
• For 1 million spheres, Pulsar runs in under 33ms forward and 11ms backward through its full Python interface, with CUDA times of 19ms and 4.7ms.These measurements use an NVIDIA RTX 2080 GPU at 1000 × 1000 resolution.Source: Table 2 · p. 6

Ease of Use

• Pulsar exposes scenes and camera parameters in PyTorch-compatible forms, allowing rendered tensors to feed neural shading or losses registered automatically with autograd.The paper illustrates a complete executable example for a ten-sphere scene.Source: §4 · p. 6

5.1. 3D Reconstruction

• Pulsar scales from silhouette reconstruction without topology-aware losses to high-resolution head reconstruction using hundreds of thousands of spheres.The silhouette experiment uses only a photometric ℓ1-loss, while the head model starts from 400k spheres and is refined to a 130k-sphere model.Source: §5.1 · pp. 6–7
• Pulsar completes illumination optimization in 31 seconds versus 1168 seconds for DSS, making it more than one order of magnitude faster in this experiment.Pulsar converges in 300 steps, whereas DSS requires 477 steps and more complex losses.Source: §5.1 · p. 7, Figure 4 · p. 7
• Neural shading combines the learned head geometry with Pix2PixHD [45], achieving 30+ FPS for geometry projection while shading takes 37 ms.With more than 320 training images, the reported perceptual difference between training and validation results is hardly visible.Source: §5.1 · p. 7

5.2. Neural Rendering

• Pulsar reconstructs and synthesizes high-resolution NeRF scenes with millions of spheres, including 2.1M spheres for ‘flower’ and 2.6M for ‘fern’.The ‘flower’ optimization completes in 20 minutes on an NVIDIA V100 GPU, while the ‘fern’ reconstruction uses optimization across three scales.Source: §5.2 · p. 8, Figure 5 · p. 8
• A shared per-pixel fully-connected shading model captures viewpoint-dependent effects in the NeRF ‘chair’ scene from 5.5M randomly initialized spheres.The model is conditioned on per-pixel view direction, and gradients are back-propagated through shading to optimize channel information.Source: §5.2 · p. 8, Figure 5 · p. 8

• Gradients for sphere positions and radii become difficult to compute when spheres project to fewer than a few pixels.Pulsar limits gradients for these spheres to feature channels and opacity, allowing opacity optimization to prune them and preventing noisy geometric gradients.Source: §6 · p. 8
• Pulsar is modular within PyTorch but not currently programmable because its optimized CUDA kernels embed the blending function and symbolic derivative.Changing the blending function requires explicit CUDA-kernel modification, which the authors describe as time consuming and error prone.Source: §6 · p. 8

7. Conclusion

• Pulsar is presented as an efficient sphere-based differentiable renderer that delivers orders-of-magnitude speedups and supports real-time optimization with millions of spheres.The conclusion emphasizes applications from 3D reconstruction to general neural rendering, alongside modularity, open-source availability, and PyTorch integration.Source: §7 · p. 8

A. Prelude

• The scene representation contains spheres with learned positions, feature vectors, radii, and opacities, and the renderer maps them to a feature image using camera parameters.The mapping is differentiable with respect to camera rotation, translation, focal length, and sensor size.Source: A. Prelude · p. 9
• Pulsar’s blending function combines sphere channel information along each camera ray using position, radius, and opacity.The appendix introduces this function as the differentiable aggregation operation for each pixel’s ray.Source: A. Prelude · p. 9
• Normalized device depth and orthogonal ray-to-sphere distance determine the geometric terms used for gradient computation.Position gradients are provided only for spheres with more than three projected pixels because smaller spheres produce numerically unstable gradients.Source: A. Prelude · p. 9
• The proposed aggregation function is commutative with respect to sphere order.This order independence is identified as important for the subsequent renderer implementation.Source: A. Prelude · p. 9

B. Data-parallel Implementation

• GPU efficiency depends on matching parallel code and memory-access patterns to SIMD execution and shared-memory behavior.The authors note that experimentation is needed because efficient solutions can be non-intuitive on these architectures.Source: B. Data-parallel Implementation · p. 9
• The CUDA/PyTorch renderer targets GPU efficiency through parallelization choices, early sphere pruning, and early stopping along rays.The implementation is written in C++ and CUDA C as a PyTorch extension.Source: B. Data-parallel Implementation · p. 9

B.1. The forward pass

• The forward pass chooses pixel-based parallelization because sphere-based writes cause memory-access collisions that require mutex protection and hurt runtime.Pixel parallelization instead requires reducing candidate spheres by exploiting spatial closeness between pixels.Source: §B.1 · p. 9
• Rendering is split into finding relevant spheres for each pixel group and drawing those spheres, with both stages tightly coupled to minimize memory accesses.A separate sphere-parallel kernel computes each sphere’s projected enclosing rectangle before the pixel-oriented stages.Source: §B.1 · p. 10

B.1.1 Step 0: Enclosing Rectangle Calculation

• Sphere projections replace acceleration structures for finding sensor-plane bounds under both orthogonal and pinhole cameras.Pinhole projection requires somewhat more computation, but trigonometric functions can be avoided through identities.Source: §B.1.1 · p. 10
• Every sphere is rendered with at least one pixel so numerical boundary errors do not cause spheres to lose gradient signals.The design stores intersection and draw information in separate data structures for subsequent processing.Source: §B.1.1 · p. 10
• Intersection and draw data for 1,000,000 spheres are stored in 0.22 ms, followed by depth sorting that takes 3.2 ms.Sorting enables early termination once farther spheres cannot make a notable contribution.Source: §B.1.1 · p. 10

B.1.2 Step 1: Intersection Search

• The renderer divides the image into nine parts, selects relevant spheres for each region, and continues with coalesced block- and warp-level processing.The selected spheres remain sorted by earliest possible intersection depth.Source: §B.1.2 · pp. 10–11
• Using 16 × 16 blocks improves intersection-search performance up to the 256-thread hardware limit, suggesting favorable scaling on future GPUs.Coalesced loading and shared-memory buffering support the search, while block-wide voting can terminate processing when pixels are finished.Source: §B.1.2 · p. 11

B.1.3 Step 2: the Draw Operation

• Each pixel thread draws loaded spheres independently, avoiding write conflicts while tracking the closest five intersections for the backward pass.A contribution threshold can terminate ray processing early when farther spheres cannot materially affect pixel colors.Source: §B.1.3 · p. 11

B.1.4 Preparing for the Backward Pass

• The forward pass stores each pixel’s normalization factor and the intersection depths and IDs of its closest five spheres.This buffer supplies the information required for the optional backward pass.Source: §B.1.4 · p. 11
• Reusing the forward-pass buffer is substantially faster than alternative backward strategies because it skips intersection search and permits unsynchronized per-thread memory access.The trade-off is the cost of writing and loading the backward-information buffer.Source: §B.1.4 · p. 11

B.2. The Backward Pass

• Pixel-wise backward parallelization outperforms sphere-wise processing and remains robust when spheres occupy large image regions.Sphere-wise processing would avoid synchronization but require each thread to traverse all pixels affected by a sphere.Source: §B.2 · p. 11
• Camera-gradient accumulation is performed sphere-wise followed by device-wide reduction, reducing runtime to 0.6 ms for 1,000,000 spheres.Direct synchronization for camera gradients creates too much memory pressure because every sphere contributes at every pixel.Source: §B.2 · p. 11
• The backward implementation is robust across settings, nearly independent of sphere size, and scales with image resolution and sphere count.Additional normalization makes gradients better suited for optimization.Source: §B.2 · p. 11
• Sphere gradients are averaged over contributing pixels, while camera gradients are normalized by projected sphere area and 1 × 10^-3.These normalizations compensate for sphere-size effects and improve stable gradient-descent loss reduction.Source: §B.2 · p. 11
• Gradient computation runs only when requested by PyTorch autodiff, contributing to the renderer’s scaling behavior.This conditional execution avoids calculating gradients that the surrounding optimization does not require.Source: §B.2 · p. 12