PaperlayerView exact revision on arXiv

Source-linked AI summary

Adaptively Placed Multi-Grid Scene Representation Networks for Large-Scale Data Visualization

Skylar Wolfgang Wurster, Tianyu Xiong, Han-Wei Shen, Hanqi Guo, Tom Peterka

arXiv:2308.02494v3eess.IVcs.CVcs.GR

TL;DR

Scientific SRNs do not adapt their parameter allocation to complex volume regions, and large-scale training can incur out-of-core I/O overhead. The paper proposes adaptive feature grids and parallel domain decomposition; APMGSRN outperforms other state-of-the-art models at similar sizes in data and image space, while its renderer is slower than contemporary work.

  • Problem

    State-of-the-art scientific SRNs do not adapt network resources to volume complexity, and single-GPU out-of-core training adds I/O overhead for large-scale data.

    Source: §1 · p. 1

  • Method

    APMGSRN learns placements for multiple feature grids, while domain decomposition trains SRNs in parallel on separate volume bricks.

    Source: §1 · p. 1

  • Results

    APMGSRN outperforms state-of-the-art models at similar model sizes in both data space and image space.

    Source: §9 · p. 9

  • Takeaways & Limitations

    The model supports efficient volume rendering in the compression domain without decompressing data to its original size.

    Source: §9 · p. 9

    The neural renderer is slower than contemporary work.

    Source: §8 · p. 9

Abstract

from arXiv · show

Scene representation networks (SRNs) have been recently proposed for compression and visualization of scientific data. However, state-of-the-art SRNs do not adapt the allocation of available network parameters to the complex features found in scientific data, leading to a loss in reconstruction quality. We address this shortcoming with an adaptively placed multi-grid SRN (APMGSRN) and propose a domain decomposition training and inference technique for accelerated parallel training on multi-GPU systems. We also release an open-source neural volume rendering application that allows plug-and-play rendering with any PyTorch-based SRN. Our proposed APMGSRN architecture uses multiple spatially adaptive feature grids that learn where to be placed within the domain to dynamically allocate more neural network resources where error is high in the volume, improving state-of-the-art reconstruction accuracy of SRNs for scientific data without requiring expensive octree refining, pruning, and traversal like previous adaptive models. In our domain decomposition approach for representing large-scale data, we train an set of APMGSRNs in parallel on separate bricks of the volume to reduce training time while avoiding overhead necessary for an out-of-core solution for volumes too large to fit in GPU memory. After training, the lightweight SRNs are used for realtime neural volume rendering in our open-source renderer, where arbitrary view angles and transfer functions can be explored. A copy of this paper, all code, all models used in our experiments, and all supplemental materials and videos are available at https://github.com/skywolf829/APMGSRN.

1 INTRODUCTION

Scientific SRNs lack adaptive parameter allocation, while large-volume training can incur out-of-core I/O overhead. The paper introduces adaptive feature grids and parallel domain decomposition to address these limits.

1 INTRODUCTION

  • Existing SRNs assume uniform volume complexity, while tree-based adaptive models add storage and computation overhead and single-GPU out-of-core training adds I/O overhead.Tree structures are not directly learnable and scale poorly with dimensionality.Source: §1 · p. 1
  • APMGSRN uses spatially adaptive feature grids and a density loss to place resources in higher-error regions without tree searching or pruning.A differentiable flat-top Gaussian approximates feature density, and delayed-start and early-stopping updates train the grid transformations.Source: §1 · pp. 1–2
  • Domain decomposition trains one SRN per volume brick in parallel across available GPUs, avoiding out-of-core sampling overhead.A hash function maps query coordinates to the model for the corresponding spatial domain during inference.Source: §1 · pp. 1–2
  • A 450 GB volume was fit in under 7 minutes on 8 GPUs, and domain decomposition improved reconstruction accuracy over a single model with the same total parameter count.The paper evaluates datasets from 32 MB to 450 GB and releases an open-source renderer and code.Source: §1 · pp. 1–2

2 RELATED WORKS

Prior SRN work includes fully connected and grid-based encoders, with adaptive tree structures proposed to allocate resources spatially. The paper positions APMGSRN as a grid-based alternative to tree-based adaptivity for scientific volumes.

2.1 Scene representation networks

  • Fully connected SRNs map coordinates to values using layers and activations, while grid-based SRNs retrieve features from encodings for decoding.Fully connected models include SIREN and NeRF; grid-based approaches include fVSRN and multi-level hash-grid NGP.Source: §2.1 · p. 2
  • Block-NeRF also represents disjoint spatial blocks with multiple models, but uses overlapping regions centered on learned geometry rather than rectangular bricks aligned to voxel data.The paper also reviews SRN uses in image, distance-field, radiance-field, and scientific scalar-field representation.Source: §2.1 · p. 2
  • Adaptive approaches such as ACORN, NSVF, and NGLOD use quadtrees or octrees that require costly refinement, pruning, storage, and search.The paper notes that tree-based space-skipping may ignore relevant regions in scientific volumes, where surfaces vary with the isovalue.Source: §2.1 · p. 3
  • APMGSRN is described as an adaptive fVSRN variant that splits one highly parameterized grid into movable, smaller feature grids focused on high-error regions.Unlike tree-based methods, its adaptivity is driven by reconstruction error rather than surfaces to refine toward.Source: §2.1 · p. 3

3 OVERVIEW

The approach has two components: an adaptive SRN architecture and a domain decomposition method for training and inference on large-scale data.

3 OVERVIEW

  • APMGSRN learns the spatial locations of multiple feature grids within a volume.The architecture is detailed in Section 4.Source: §3 · p. 3
  • Domain decomposition divides a volume into a 3D grid of networks, each trained on its local region.The method is detailed in Section 5.Source: §3 · p. 3
  • The two components are an APMGSRN architecture and a domain decomposition training and inference method for large-scale data.Source: §3 · p. 3

NETWORK

APMGSRN replaces tree-based resource allocation with movable feature grids whose encoded features are decoded by a shallow MLP. The encoder transforms coordinates into each grid’s local space, interpolates features, and concatenates them; the decoder scales outputs to the volume’s value range.

NETWORK

  • Tree-based adaptive SRNs incur training and storage overhead, while APMGSRN uses learnable spatially placed feature grids and a shallow MLP for coordinate encoding and decoding.The grids can fit data complexity while avoiding tree structures.Source: NETWORK · p. 3

4.1 APMGSRN architecture

  • Each APMGSRN maps normalized spatial coordinates to scalar values through an encoder and a small fully connected decoder.The encoder uses multiple grids, each with a learnable transformation matrix.Source: §4.1 · p. 3, §4.1.1 · p. 3

4.1.1 Adaptively placed multi-grid encoder

  • The encoder transforms a global coordinate into each grid’s local space, trilinearly interpolates its features, and concatenates the resulting vectors.Grid extents are assumed to be [−1,1]3; queries outside a grid return zero features, and concatenation yields C·M features.Source: §4.1.1 · pp. 3–4

4.1.2 Decoding

  • The decoder is a two-layer, 64-neuron-per-layer MLP, and output scaling maps its values from [0,1] to the volume’s saved minimum-to-maximum range.The paper does not test z-score, log, or exponential scaling.Source: §4.1.2 · p. 4

4.2 Learning feature grid transformation matrices

APMGSRN learns feature-grid transformations to place more network capacity in high-error regions. A feature density loss is needed because reconstruction loss alone cannot move grids toward high-error regions outside their current extents.

4.2 Learning feature grid transformation matrices

  • Reconstruction loss cannot move a feature grid toward high-error regions outside its extent because outside-grid queries return zero features and provide no transformation-matrix gradients.The density loss supplies gradients for grid transformations where reconstruction loss does not.Source: §4.2 · p. 4
  • Feature density measures the average number of grid features per unit volume, and its loss matches current density to a target that rises in relatively high-error regions.The target is derived from relative error, and the density loss is a KL divergence used to update grid transformations.Source: §4.2 · pp. 4–5
  • A grid-overlap count misrepresents density because equal-resolution grids with smaller global extents have denser features; the formulation accounts for grid scale, shear, and rotation through a determinant.As a grid becomes denser in the global domain, the determinant increases to represent its feature density.Source: §4.2 · p. 4
  • A flat-top Gaussian approximation makes feature density differentiable everywhere, allowing grid transformations to receive gradients even for points outside a grid.The flat-top strength p=10 was experimentally sufficient to approximate a box shape without excessively strong gradients on the ramps.Source: §4.2 · p. 4

4.3 APMGSRN training

APMGSRN training separates reconstruction updates from grid-position updates, using density loss to move the transformations and reconstruction loss to fit grid values and decoder weights. It delays and eventually stops transformation updates, and initializes grids near the global domain.

4.3.2 Training routine.

  • Reconstruction loss updates feature-grid values and decoder weights, while density loss alone updates transformation matrices; both updates are not performed every iteration.The two-part routine limits transformation updates because performing them every iteration could slow training and negatively affect reconstruction quality.Source: §4.3.2 · p. 5
  • Transformation matrices remain frozen for the first 500 iterations, then receive density-loss updates after the reconstruction-error distribution becomes more informative.The 500-iteration delay is reported for the paper’s learning rate and network sizes.Source: §4.3.2 · p. 5
  • Transformation updates stop when the 1000-iteration moving average of density loss fails to improve by 0.01%, or after 80% of training steps.Fixing grid positions lets other network parameters fine-tune against static grids and avoids spending time on transformations that may have converged.Source: §4.3.2 · p. 5
  • A single APMGSRN trains for at most 4 minutes at 50k iterations, with grids often converging quickly enough to reduce training time to 40 seconds.Source: §4.3.2 · p. 5

4.3.3 Initialization

  • Transformation matrices initialize grids near the global domain with small random scale, shear, rotation, and translation so every relevant entry can receive gradients.Grid scales are sampled from N(1,0.05), and remaining entries in the first three rows from N(0,0.05); feature grids and decoder weights use separate initialization schemes.Source: §4.3.3 · p. 5

5 DOMAIN DECOMPOSITION SCENE REPRESENTATION

Domain decomposition represents a large volume with separate SRNs assigned to spatial bricks, avoiding out-of-core sampling and communication among models. The paper describes brick partitioning, parallel training, and spatial hashing for inference, while noting that its current inference implementation evaluates all networks.

5 DOMAIN DECOMPOSITION SCENE REPRESENTATION

  • A single model may be infeasible when complex, high-resolution data demand larger networks or when the volume exceeds GPU memory, making random CPU sampling costly and out-of-core sampling a fallback.Source: §5 · p. 5
  • Separate brick-specific SRNs form a model-parallel representation of the volume, with no communication required during training or inference.The approach is proposed instead of a complicated out-of-core sampling method.Source: §5 · p. 5

5.1 Data partitioning

  • The volume is partitioned using a grid of networks, with the number of bricks per axis chosen so each assigned volume has roughly the feature grids’ aspect ratio.For feature grids of resolution 32^3, the paper recommends roughly cube-shaped assigned volumes.Source: §5.1 · p. 6
  • Overlapping ghost cells between neighboring networks tend to reduce boundary seams but do not guarantee their removal.Experiments use between 1 and 16 ghost cells.Source: §5.1 · p. 6

5.2 Domain decomposition training

  • Available GPUs train brick-specific models in parallel, loading only each model’s assigned extent; this reduces I/O overhead with formats supporting parallel access and cropping.Jobs are issued to GPUs as they become available.Source: §5.2 · p. 6

5.3 Domain decomposition inference

  • Inference hashes each global query coordinate to its brick’s model index and scales the coordinate to that model’s local domain.The implementation loops over all networks for evaluations; custom CUDA engineering could significantly speed inference, but is left to future work.Source: §5.3 · p. 6

6 NEURAL VOLUME RENDERING

The authors provide a Python/PyTorch neural volume renderer for comparing SRNs and raw volume rendering, with support for arbitrary transfer functions and view directions. Progressive rendering provides interactive approximations and a user-selectable trade-off between interactivity and full-frame rendering time.

6 NEURAL VOLUME RENDERING

  • The Python/PyTorch renderer supports APMGSRN, fVSRN, hash-grid NGP, and raw volume rendering, and accepts other PyTorch coordinate-to-density models with minimal reconfiguration.It supports arbitrary transfer functions and view directions and favors ease of use over raw performance.Source: §6 · p. 6
  • Progressive checkerboard rendering displays bilinearly upscaled approximations between passes and maintains interactive frame rates of 30–400 fps.Larger batch sizes slow interaction but speed full-frame rendering; smaller batches do the reverse.Source: §6 · p. 6

7 EXPERIMENTS

The experiments compare APMGSRN’s reconstruction, compression, domain-decomposition, and rendering performance across scientific datasets. They find reconstruction gains in most settings, alongside trade-offs in training speed, compression, ghost-cell use, and rendering performance.

7.1 Scene representation network comparison

  • APMGSRN ranks first in PSNR in all but three dataset-and-size settings, with gains exceeding 2 dB on sparse plume, Nyx, and supernova data at the largest size.NGP outperforms APMGSRN on the highly sparse asteroid dataset.Source: §7.1 · p. 7
  • The default and 20%-larger grid initializations produce similar final grids and reconstruction quality, while small randomly placed grids perform much worse.Grids outside the data receive near-zero gradients for changing transformation parameters.Source: §7.1 · p. 7, Figure 4 · p. 7
  • APMGSRN exceeds TTHRESH and SZ3 in compression ratio only at 10000× data reduction below 30 dB PSNR; its GPU decoding reaches about 500 million points per second.For 250 GB rotstrat, APMGSRN models achieve 41.32 and 44.29 dB PSNR at 4791× and 1432× compression, respectively; SZ3 achieves 20.60 dB at 509×.Source: §7.1 · p. 8, Figure 5 · p. 8

7.2 domain decomposition network evaluation

  • At equal storage size, domain-decomposition models outperform single models on every tested dataset, while serial training may take longer on one GPU.On 8 GPUs, rotstrat training takes 7 minutes including data I/O.Source: §7.2 · p. 8
  • Using 16 rather than 1 ghost cell removes the visible seam but drops PSNR by 1 dB; the authors recommend few cells for rendering and 16 for isosurface extraction.The larger overlap increases the volume learned by each network from 256^3 to 288^3, reducing the set of networks’ learning capacity.Source: §7.2 · p. 8, Figure 6 · p. 8

7.3 Neural volume rendering

  • On 64 MB Supernova models, APMGSRN renders slowest but has higher image PSNR and SSIM than the other neural representations; NGP renders fastest among the networks.The raw volume renders fastest but requires GPU memory and cannot fit on the test machine above 11 GB.Source: §7.3 · p. 8, Figure 7 · p. 9
  • For 250 GB rotstrat, the 183 MB neural model renders at 30.70 dB PSNR, versus 22.24 dB for similarly sized downsampled data, and captures features the downsampled data misses.Both representations were rendered into 8000^2 images with the same visual mapping parameters.Source: §7.3 · pp. 8–9

8 LIMITATIONS

APMGSRN has two stated efficiency limitations: increasing feature-grid count raises memory use and slows training, while the renderer is slower than contemporary reported work.

8 LIMITATIONS

  • Increasing the number of feature grids raises memory use and slows training because coordinate transformations and interpolation are performed for each grid.The transformation stage allocates memory proportional to query-coordinate memory times the number of grids.Source: §8 · p. 9
  • The PyTorch-compatible neural renderer is slower than contemporary reported renderers.The paper notes that code optimization could improve rendering speed while preserving plug-in flexibility.Source: §8 · p. 9

9 CONCLUSION AND FUTURE WORK

The conclusion reports that APMGSRN improves reconstruction over state-of-the-art models at similar model sizes and supports large-scale representation through domain decomposition. It also identifies compression trade-offs and future directions for reducing artifacts and improving efficiency.

9 CONCLUSION AND FUTURE WORK

  • Domain decomposition divides a volume into chunks represented by multiple networks, enabling training and inference on data too large to fit in-core.The paper describes this approach as a way to train large-scale data.Source: §9 · p. 9
  • Figure 8 compares renders of the original rotstrat data, the medium model, and an 11×-subsampled version with similar storage footprint.The images use the same visual mapping parameters, and the listed PSNR/SSIM metrics cover the full image.Source: Figure 8 · p. 9
  • The renderer interface supports loading models or NetCDF data, choosing colormaps, editing opacity transfer functions, and adjusting batch size and samples per ray.Larger query batches reduce total frame time but require more GPU memory and slow interactivity.Source: Figure 9 · p. 9
  • APMGSRN outperforms other state-of-the-art models at similar model sizes in both data-space and image-space reconstruction, though it compresses less than TTHRESH.It still achieves high compression with volume rendering in the compression domain without decompressing to original data size.Source: §9 · p. 9
  • Future work includes reducing boundary artifacts and improving efficiency, extending grids to temporal data, and using alternative grid encodings or non-uniform data types.The paper also identifies support for neural volume rendering in mature open-source software as a future direction.Source: §9 · p. 9

Related papers

Loading 2308.02494v3…