Source-linked AI summary
ABCD: Alpha-Composited Block Coordinate Descent: Constant-VRAM Training for Large Radiance Fields
Ka Heng Shiu, Kartic Subr
TL;DR
Training increasingly large radiance-field scenes can exceed modern GPUs’ memory limits, motivating ABCD, an out-of-core method based on block coordinate descent and alpha compositing. ABCD preserves reconstruction quality substantially better than its compositing ablation while making memory effectively independent of total scene extent.
Problem
Training increasingly large scenes often reaches modern GPUs’ memory limits and may require multi-GPU distributed setups.
Method
ABCD trains alpha-composited radiance fields by applying block coordinate descent over spatial partitions and collapsing inactive regions into foreground/background images.
Results
ABCD reaches an average PSNR of 24.73, only 3.1% below the reference, whereas removing compositing causes a 36.9% degradation.
Takeaways & Limitations
ABCD makes memory effectively independent of total scene extent and targets settings where conventional training may be infeasible on available hardware.
Takeaways & Limitations
The experiments were limited by time and engineering effort, including incomplete tuning of densification.
Abstract
from arXiv · showhide
We present ABCD (Alpha-Composited Block Coordinate Descent), an out-of-core training framework for alpha-composited radiance fields, instantiated here for 3D Gaussian Splatting. Our method reformulates training as block coordinate descent over spatial partitions: only one block of parameters is active at a time, while all others are frozen. By exploiting the associativity of alpha blending, these inactive regions can be pre-rendered and collapsed into foreground and background RGBA images. As a result, for fixed partition size and image resolution, peak VRAM becomes O(1) with respect to total scene extent, rather than growing with full scene size. This enables GPUs with limited memory to train scenes that would otherwise not fit in core. In experiments, our method closely preserves the reconstruction quality of 3DGS, with less than 5% PSNR degradation, while ABCD with compositing ablated suffers roughly 40% degradation. Our code can be found at https://github.com/shiukaheng/abcd
1 Introduction
Large radiance-field scenes can exceed affordable GPU memory, while independently training spatial partitions introduces artifacts when views span partition boundaries. ABCD addresses this gap with out-of-core block coordinate descent for alpha-composited radiance fields.
- Large-scene radiance-field training often reaches GPU memory limits and may require prohibitively expensive multi-GPU setups.
- Independent partition training can create severe artifacts when images observe geometry spanning multiple partitions.Each partition may compensate for content missing from neighboring partitions.
- Partitioning improvements such as overlaps or auxiliary boundary primitives still depend on decomposition quality.
- 3DGS-specific LOD methods reduce memory but are not general to arbitrary radiance-field formulations.
- ABCD is presented as a principled out-of-core training method for alpha-composited radiance fields.Its formulation uses block coordinate descent over spatial partitions and is demonstrated on 3D Gaussian Splatting.
2 Method
ABCD activates one spatial parameter block while freezing the others, then uses alpha compositing to represent inactive regions as cached foreground and background images. This keeps GPU memory focused on the active block and required image pair during optimization.
- The full scene parameters are partitioned into spatial blocks, with one block selected as active while the remaining parameters stay fixed.
- Frozen parameters are separated into regions entirely in front of or behind the active partition.The method denotes these sets as Θfg and Θbg.
- By alpha-compositing associativity and fixed depth ordering, inactive contributions are pre-accumulated into flattened foreground and background RGBA images.
- The system stores cached camera–partition renders on disk, while GPU memory contains the active parameters and required foreground/background images instead of the full parameter set.Streaming can reduce GPU usage further by processing a single image pair at a time.
- During each update, only the active partition is rendered, composited with cached images, and differentiated through the active parameters.After T gradient steps, cached renders for that partition are refreshed before proceeding to the next block.
3 Results
ABCD preserves reconstruction quality close to 3DGS while retaining partitioned-training memory behavior, but increases system RAM and runtime. Its GPU-memory advantage is modest on the small evaluated scenes, where optimizer and image overheads dominate.
- Reconstruction quality: ABCD reaches an average PSNR of 24.73, only 3.1% below the 3DGS reference, while compositing ablation causes a 36.9% degradation.Across both garden and kitchen scenes, the ablated variant falls from 25.52 to 16.10 PSNR.
- Reconstruction quality: ABCD reaches SSIM 0.707, only 3.0% below 3DGS, whereas compositing ablation reduces average SSIM from 0.728 to 0.430.The qualitative comparison likewise reports that full ABCD closely matches 3DGS, while ablation introduces severe boundary and consistency artifacts.
- Optimization and memory: ABCD follows a training trajectory similar to 3DGS and recovers most of the ablated method’s lost reconstruction quality while retaining nearly identical active Gaussian residency.The results attribute the preserved memory behavior to partitioned training with low active Gaussian residency.
- Optimization and memory: For fixed partition size and image resolution, peak VRAM averages 0.785 GB for ABCD, 9.0% above ablation but 15.1% below 3DGS.ABCD’s VRAM depends on the active partition and one foreground/background image pair rather than total scene extent.
- Resource trade-offs: The small evaluated scenes mask ABCD’s asymptotic VRAM benefits because optimizer state and image-related overheads dominate total GPU memory consumption.The evaluation uses garden and kitchen scenes with approximately 200 training cameras and a regular spatial grid.
- Resource trade-offs: ABCD uses 21.38 GB system RAM, 193.2% above compositing ablation, because cached rendered images are stored for cameras observing active partitions.These cached images can be stored on disk and streamed, with transfer cost amortized across multiple optimization steps on a shard.
4 Limitations and Future Work
The experiments remain limited in scene scale, engineering and theoretical validation, while several implementation choices constrain robustness and efficiency. Future work targets larger datasets, better visibility handling, and distributed extensions.
- Current experiments evaluate only relatively small scenes and did not fully tune densification and training hyperparameters for 3D Gaussian Splatting.
- 32?
- The asymptotic memory savings are masked when optimizer state, image storage, and rasterization overheads dominate GPU memory, but should become more pronounced in larger scenes.
- Training a partition requires cached render pairs for observing cameras, and frustum culling can still make this storage grow with scene scale.
- Visibility thresholds and dynamic occlusion culling could reduce memory and storage costs as distant partitions become negligible in image area.
- Convergence behavior for spatially grouped Gaussian Splatting variables remains open, and mean-based assignment can mishandle Gaussians whose covariance crosses partition boundaries.
- Future work should test substantially larger real-world datasets and distributed variants that exchange rendered partition images intermittently across compute nodes.