Fast Neural Representations for Direct Volume Rendering

2. Related Work

• Earlier neural approaches use super-resolution for spatial or temporal data [ZHW∗17,HW20,GYH∗20, HW19], while SRNs map domain positions directly to field values [LJLB21,WSG∗21].Unlike grid-based super-resolution, SRNs allow random access and camera movement during testing; neurcomp adapts SRNs to predict density samples.Source: §1 · p. 1, §2 · p. 3

1. Introduction

• Generic deep-network evaluation writes intermediate layer states to global memory, making V-SRN rendering memory-bound, non-interactive, and slow to train.Neurcomp samples density directly from its compact latent representation, but calls a deep network at every view-ray sample.Source: §1 · p. 1

1.1. Contribution

• fV-SRN combines a volumetric latent-feature grid with a compact network that uses GPU TensorCores and on-chip memory for faster training and rendering.It renders directly from the compressed representation without additional rendering-time memory.Source: §1.1 · p. 2
• Against neurcomp, fV-SRN is significantly faster at similar compression rates; against TThresh, it achieves similar compression rates with significantly lower reconstruction times, and it significantly outperforms cudaCompress in compression ratio at similar decoding speed.Source: §1.1 · p. 2
• The proposed extensions jointly predict scalar values, gradients, and curvature, and use latent-space interpolation for feature-preserving temporal reconstruction at arbitrary temporal resolution.This removes the restriction of methods such as TSR and STNet to interpolation at predefined timesteps.Source: §1.1 · p. 2

2. Related Work

• The paper positions fV-SRN among lossy scalar-field compression methods, including quantization, curve fitting, transform coding, and tensor decomposition.It targets high compression rates associated with offline schemes while aiming to render large volumes from the compressed representation within a second.Source: §2 · p. 3
• Surface-focused SRN techniques cannot directly transfer to direct volume rendering, where surfaces may be absent or change with interactively selected isocontours.The paper identifies potential future transfer of TensorCore kernels and aliasing-reduction techniques such as Mip-NeRF [BMT∗21].Source: §2 · p. 3

3. Scene Representation Networks

• SRNs [MST∗20,TSM∗20] encode a field with a fully connected network that maps each 3D position to its density or color value.The network’s trainable weights and biases define the mapping from input position to output.Source: §3 · pp. 3–4
• Because each position is processed independently, SRNs can decode only the positions needed along a ray, processing thousands in parallel; neurcomp trains on world-space position–density pairs.This follows prior SRN approaches [CZ19,MON∗19,PFS∗19].Source: §3 · p. 4
• Fourier features address SRNs’ difficulty representing high-frequency data by lifting positions to emphasize output variation; the authors found fixed powers-of-two features improved learning, but random features added no observed quality improvement.The fixed Fourier matrix is not trainable; random features use a dataset-dependent σ.Source: §3 · p. 4

4. Fast Volumetric SRN (fV-SRN)

• Fusing the network into one CUDA kernel avoids global-memory transfers of intermediate layer results, addressing repeated network evaluation as a rendering bottleneck.For 100 ray steps and a seven-layer SRN, separate layer evaluations entail 700 invocations and global-memory reads and writes.Source: §4 · p. 4

4.1. Custom Inference via TensorCores

• TensorCore inference maps one thread to one ray and tiles layer matrices into 16 × 16 blocks, while hidden-layer weights, biases, and outputs must fit in shared memory.The implementation accommodates TensorCore requirements for matrix dimensions, memory access, and warp-synchronous execution.Source: §4.1 · p. 4
• Custom CUDA TensorCore inference is up to 16.8× faster than 32-bit PyTorch and 9.8× faster than 16-bit PyTorch.These comparisons use the same V-SRN network architecture.Source: §4 · p. 4, §4.1 · p. 5

4.2. Volumetric Latent Grid

• The dense latent grid is trilinearly interpolated and jointly trained with the network; 8-bit grid entries reduce its memory footprint to one quarter, with roughly 5% slower rendering and up to 2% lower reference SSIM and LPIPS statistics.The grid compensates for reduced network expressive power and suits direct volume rendering, where transfer-function mappings may change after training.Source: §4.2 · p. 5
• The latent grid lets fV-SRN use networks as small as two layers with 32 channels while maintaining V-SRN reconstruction quality.The selected speed-quality-compression compromise is a four-layer, 32-channel network with a grid of resolution R = 32 and F = 16 features.Source: §4.2 · p. 5
• Compared with neurcomp, fV-SRN achieves up to 9.8× faster training and 165× faster rendering.The latent grid is stored in CUDA textures to use hardware-supported trilinear interpolation.Source: §4.2 · p. 5

5.1. Performance Evaluation

• The 6-layer, 48-channel network achieved speedups of 16.8× over 32-bit PyTorch and 9.8× over 16-bit PyTorch; larger networks still achieved 1.9× and 1.5× speedups, respectively.For very small networks, speedups fell to approximately 11× and 8×; renderings at 5–10 FPS were possible only with small networks.Source: §5.1 · p. 6, Figure 4 · p. 6

5.2. Latent Grid

• Higher latent-grid resolution sharply improved reconstruction quality; at resolutions of 32^3 or higher, network choice had limited effect, and four layers with 32 channels sufficed.Combining the latent grid with the SRN preserved small-scale structures better than a same-memory low-pass density grid; fV-SRN training was up to 9.8× faster than V-SRN with the same parameter count.Source: §5.2 · pp. 6–7, Figure 5 · p. 6, Figure 6 · p. 7

5.3. Fourier Features

• Fourier features generally improved reconstruction quality, and “NeRF” features outperformed Tancik et al.’s random Fourier features with this network design.The study compared “NeRF” features [MST∗20], Tancik et al.’s features [TSM∗20] at multiple standard deviations, and no Fourier features.Source: §5.3 · p. 7, Figure 7 · p. 7

5.4. Density vs. Color Prediction

• Color prediction matched density prediction for two transfer functions but performed considerably worse for a transfer function with two narrow peaks.Adaptive resampling based on prediction error improved the color network’s LPIPS score for the third transfer function from 0.113 to 0.039, but sharp transfer functions remained difficult; unlike density prediction, color prediction requires retraining when the transfer function changes.Source: §5.4 · pp. 7–8

6. Gradient Prediction and Higher-Order Derivatives

• Jointly predicting gradients improves rendering performance by ≈12× over finite differences and 4−5× over the adjoint method, while reducing quality only slightly.The network predicts density and three gradient components within the same weights and latent-grid size, increasing output dimensionality from one to four.Source: §6 · p. 8
• Central-difference FD avoids voxel-border discontinuities but requires seven network evaluations; Adjoint uses backpropagation through the reconstruction network.Gradients enable TF-based opacity and color selection and provide isosurface normals for lighting.Source: §6 · p. 8
• For higher-order derivatives, FD and Adjoint become increasingly costly, motivating joint prediction of density, gradients, and curvature.The Kindlmann et al. finite-difference stencil uses 43 samples and would require 64 network evaluations per ray sample; Adjoint requires an additional pass per Hessian row. Initial curvature-rendering results compare predicted and reference curvature on an implicit dataset and CT scan.Source: §6 · p. 8, Figure 10 · p. 8

Performance and Quality Comparison

• fV-SRN, neurcomp, and TThresh [BRLP19] are almost on-par in PSNR and SSIM; network methods slightly outperform TThresh at high compression, while TThresh leads at low compression.The comparison evaluates reconstruction of random locations and image rendering quality.Source: §7 · p. 9
• fV-SRN requires no additional temporal evaluation memory, unlike TThresh [BRLP19] and cudaCompress [TRAW12,TBR∗12], which can require up to 34GB and 4.7GB, respectively.TThresh and cudaCompress decode the volume before rendering; neurcomp stores hidden states during network evaluation.Source: §7 · p. 9
• For Ejecta, bricked TThresh [BRLP19] reduces memory from 34GB to around 8GB but drastically lowers its compression ratio.Each brick is compressed independently.Source: §7 · p. 9
• For Ejecta, bricked cudaCompress [TRAW12,TBR∗12] reduces memory from 4.7GB to around 700MB, increasing rendering time by roughly 13%.Bricked rendering works only for regular rendering access; random access still requires decompressing the whole volume.Source: §7 · p. 9
• cudaCompress [TRAW12,TBR∗12] produces large errors with narrow transfer functions, TThresh [BRLP19] shows slight grid artifacts, and fV-SRN and neurcomp blur at higher compression.The error patterns are shown in Fig. 11.Source: §7 · p. 9

• The method extends volumetric latent spaces over time, storing them at keyframes and linearly interpolating between them; it draws on latent-vector interpolation by Park et al. [PFS∗19] and Chen and Zhang.Training uses a subset of available timesteps.Source: §8 · p. 9
• The evaluation compares four time encodings—latent only, direct, Fourier features, and both—on ScalarFlow with a keyframe every 10th timestep.Training uses every 5th or every 2nd timestep; qualitative results cover timesteps 60 to 70.Source: §8 · p. 9
• “Latent only” and “direct” produce semantically plausible interpolation at unseen timesteps, unlike linear interpolation of the original grid at keyframes.The two fV-SRN options show no noticeable difference from each other.Source: §8 · p. 9
• Fourier time encodings (“fourier” and “both”) behave chaotically between training timesteps because discrete temporal samples leave unseen encoding ranges during generalization.Source: §8 · p. 9
• Neurcomp predicts training timesteps accurately at the same compression ratio but fails between them; a single-keyframe grid also worsens direct encoding results.The authors hypothesize that the time-interpolated latent grid acts as a regularizer.Source: §8 · p. 9
• Training on every 5th timestep takes around 3:45h, while every 2nd timestep improves interpolation quality but increases training time to almost 8 hours.Source: §8 · p. 9

• The work considers network-based gradient prediction and proposes accelerated training and adapting fV-SRN for temporal super-resolution.Source: §9 · p. 10
• fV-SRN offers an alternative to existing volume compression schemes with comparable quality and significantly improved decoding speed, or similar performance at significantly higher compression ratios.Source: §9 · p. 10
• Custom evaluation kernels and a latent grid give SRNs potential for interactive volume rendering, and accelerated training is proposed.Source: §9 · p. 10
• Latent-space interpolation can preserve features lost with traditional interpolation and enable temporal super-resolution at arbitrary temporal resolution.Source: §9 · p. 10
• Future work should investigate view-dependent level-of-detail learning to avoid losing details smoothed by classical low-pass filtering.The proposed direction is to predict values as perceived through a pixel and perform area-weighted super-sampling.Source: §9 · p. 10
• Random access to compressed samples may support flow visualization by encoding position-velocity relationships for particle or streamline tracing.This targets sparse, highly irregular accesses to compact vector-field representations.Source: §9 · p. 10
• For ensemble visualization, future work will explore multiple latent grids, Mean-SRNs, and separating ensemble from temporal information.The conclusion also proposes encoding differences from a reference and adaptive temporal resampling to reduce the training burden of time-dependent data.Source: §9 · p. 10

• It compares several activation functions in Section 1.Source: Fast Neural Representations for Direct… · p. 14
• Section 2 analyzes training the network from images rather than 3D points.Source: Fast Neural Representations for Direct… · p. 14
• Section 3 gives further details on extending the network to provide both gradients and densities.Source: Fast Neural Representations for Direct… · p. 14
• Section 4 presents two additional datasets for comparisons against baseline methods.Source: Fast Neural Representations for Direct… · p. 14

1 Activation Functions

• Snake and SnakeAlt were nearly on par, with SnakeAlt leading on two of three datasets in comparisons with ReLU and Sine [SMB∗20].The comparison also included Snake and evaluated rendered images.Source: §1 · p. 14, Table 1 · p. 14
• SnakeAlt is motivated by providing a more monotonous slope than the original Snake.Source: §1 · p. 14

2 Image-based Training

• The section frames perceptual-adaptive encoding as allocating minimal resources to task-irrelevant data and making important structures easy to distinguish.It presents screen-space SRNs as a first step toward integrating such codes with a renderer.Source: §2 · p. 15

2.1 Training Methodology

• Screen-space training predicts color and absorption directly because non-convex transfer functions make density reconstruction ill-posed.Color is constrained to [0,1] with a sigmoid and absorption to R+ with a softplus.Source: §2.1 · p. 15
• Differentiable volume rendering uses analytic inversion of blending to keep backpropagation memory constant regardless of steps per ray.This addresses the memory-intensive storage of intermediate ray values in earlier training approaches.Source: §2.1 · p. 15

2.2 Ablation Study and Results

• Screen-space training takes about 2.5 hours for 96 images at 256^2 resolution and stepsize 0.02, versus roughly 30 minutes for world-space training.The world-space run uses 500 epochs and 16 million points; the screen-space run uses 6 million pixels.Source: §2.2 · p. 16
• At ∼1-voxel stepsizes, screen- and world-space networks achieve almost equal quality, with a slight advantage for world-space training.The reported quality comparison uses a TF with sharp peaks; smoother-peak TFs behave similarly.Source: §2.2 · p. 16, Table 2 · p. 16
• At ∼4-voxel stepsizes, world-space training produces ringing from undersampling, while screen-space training avoids ringing but blurs sharp color transitions.Despite the blur, screen-space training reconstructs the field’s overall structure at coarse stepsizes.Source: §2.2 · pp. 16–17, Figure 1 · p. 17

3.1.1 Finite Differences

• Finite differences require six additional network evaluations (C = 6) and no stored hidden states (M = 0); setting the step size to one voxel avoids the large- and small-step problems described.A step that is too large can skip small features, while one that is too small can cause numerical instability from cancellation.Source: §3.1.1 · p. 18

3.1.2 Automatic Differentiation

• For four-layer fV-SRNs, the adjoint method is more advantageous than forward differentiation, though larger networks may favor forward differentiation because adjoint memory scales with layer count.Adjoint uses C = 1 additional network evaluation and M = 3 stored hidden states; forward differentiation uses C = 3 additional matrix evaluations and M = 3 stored forward states.Source: §3.1.2 · p. 19

3.1.3 Direct Gradient-Predicting SRNs

• Direct prediction jointly outputs density and its gradient, requiring retraining but no additional inference cost (C = 0, M = 0).The training loss combines density and gradient losses; gradient targets come from analytic derivatives for implicit functions or finite differences on grids.Source: §3.1.3 · p. 20

3.2 Hyperparameters and Ablation Study

• A recommended starting configuration is w = −6 (α ≈ 0.0025), L2 gradient loss, direct output, and resampling with p = 0.1 and c = False.The study found a U-shaped relationship between gradient weighting and quality, with dataset-specific optima; loss choice mattered, while output parametrization and resampling had smaller or insignificant effects. Fine-tuning α can improve results on new datasets.Source: §3.2 · p. 21, §3.3.2 · p. 24

3.3.1 Implicit Functions with Analytical Gradients

• On analytic implicit functions, direct prediction has the lowest mean gradient-magnitude error and highest directional accuracy, while the adjoint method performs worst.The authors attribute the adjoint method’s errors to numerical inaccuracies from passing inputs through twice as many computations; direct prediction also gives the smoothest results and generally the best SSIM and LPIPS, with one Marschner Lobb SSIM exception for finite differences.Source: §3.3.1 · pp. 23–24

3.3.2 Real-world Datasets

• On real-world grids, finite differences yield the best SSIM and LPIPS, while direct gradient prediction slightly reduces quality but renders around 10× faster.The reference gradients use central differences with a voxel-size step, biasing this comparison toward finite differences.Source: §3.3.2 · p. 24

3.3.3 Performance

• For density and gradient evaluation on 224 samples, direct gradient-predicting SRNs take up to 1.66× the baseline rendering time; adjoint is up to 5.62× slower, and finite differences incur up to 12.50× overhead.Source: §3.3.3 · pp. 24–25

4 Comparisons against Baseline Compression Methods – Further Results

• On Richtmyer-Meshkov (1024^3) and Miranda (512^3), fV-SRN is compared with neurcomp, TThresh, and cudaCompress [TRAW12, TBR∗12] at different compression ratios.The quantitative and qualitative results are presented in Figures 7 and 8.Source: §4 · p. 26
• Figure 8 shows visual comparisons on both datasets using examples closest to 1:1000 compression for Richtmyer-Meshkov and 1:10,000 for Miranda.The examples include the reference, fV-SRN, TThresh, and cudaCompress.Source: Figure 8 · p. 27, §4 · p. 27