PointAcc: Efficient Point Cloud Accelerator

• Point clouds represent physical objects or 3D scenes and are usually generated by sensors.Source: §1 · p. 1
• PointAcc accelerates point-cloud computation by resolving sparsity and data-movement bottlenecks.Source: Figure 1 · p. 1

2D CNNs

• Direct 3D processing yields up to 5% higher accuracy with 7× less #MACs than projecting point clouds to 2D, yet point-cloud networks run slower on existing hardware.MinkowskiUNet runs at 11.7 FPS on an RTX 2080Ti, compared with 840 FPS for ResNet50.Source: 2D CNNs · p. 1, Figure 2 · p. 1
• Physical point-cloud sparsity constrains nonzero locations, so nonzero points do not dilate during computation.Source: 2D CNNs · p. 2
• PointAcc unifies diverse mapping operations, including kernel mapping, k-nearest-neighbors, ball query, and farthest point sampling, through a ranking-based computation paradigm.This targets mapping operations unsupported by existing accelerators.Source: 2D CNNs · p. 2
• PointAcc uses configurable caching and temporal fusion of consecutive layers to reduce DRAM access by up to 6.3×.Source: 2D CNNs · p. 2
• Implemented and synthesized in TSMC 40nm, PointAcc was evaluated on 8 modern point-cloud networks across 5 datasets and achieved an order-of-magnitude average speedup over other hardware platforms.Source: 2D CNNs · p. 2

2 BACKGROUND

• Point cloud convolution multiplies each mapped input feature by its neighbor-specific weight and scatter-aggregates partial sums at the corresponding output point.Unlike image convolution, its maps must be calculated explicitly when downsampling; unlike graph convolution, different neighbors use different weights.Source: §2 · p. 2
• Point cloud convolution comprises mapping, feature movement by gather/scatter, and matrix multiplication.Convolutions are categorized as PointNet++-based or SparseConv-based; graph-based convolutions are treated as a PointNet++-based special case.Source: §2 · pp. 2–3

2.1 Mapping Operations

• Mapping finds input-output maps by constructing output points and searching for their neighbors.The map associates an input point, an output point, and a weight.Source: §2.1 · p. 3

2.1.1 Output Point Cloud Construction.

• SparseConv-based downsampling quantizes output coordinates using tensor stride, whereas PointNet++-based downsampling selects points iteratively by farthest-point sampling.Upsampling reverses the corresponding downsampling operation.Source: §2.1.1 · p. 3

2.1.2 Neighbor Search.

• SparseConv kernel mapping checks neighborhood offsets, while PointNet++ neighbor search selects top-k points by distance and ball query additionally enforces a radius.For ball query, selected points must satisfy ||p−q||2 ≤ r.Source: §2.1.2 · p. 3

2.2 MatMul Operations

• For each weight, gathered input features are multiplied to form partial sums, which are scattered to output points and aggregated.Aggregation may use max-pooling, convolution, or accumulation.Source: §2.2 · p. 3

3 MOTIVATION

• Outdoor point clouds can have density below 10^-4, while indoor scenes and objects are below 10^-2; ImageNet is 100% dense at input.Point-cloud sparsity reflects spatial distribution and physical information, constraining its pattern unlike sparsity from CNN ReLU activations.Source: §3 · p. 4
• PointNet++-based networks spend over 50% of runtime on mapping, which existing specialized accelerators do not support.On TPU, moving data to host memory for CPU mapping and gathering, then returning contiguous matrices, makes data movement 60%–90% of runtime.Source: §3 · p. 4
• Point cloud convolution can access features up to 27 times through explicit gathering and scattering, and feature memory per point can reach 16 KB.SparseConv downsampling reduces spatial resolution without usually reducing point count by 4× as in 2D CNNs; feature movement alone exceeds 50% of CPU/GPU runtime.Source: §3 · p. 4

4 ARCHITECTURE

• PointAcc uses a Mapping Unit, Memory Management Unit, and Matrix Unit to address point cloud processing challenges.The Memory Management Unit bridges the other two units by preparing data for the Matrix Unit based on Mapping Unit output.Source: §4 · p. 4
• The Memory Management Unit prepares data for the Matrix Unit using output from the Mapping Unit.Source: §4 · p. 4
• Configurable data flow across the units supports diverse point cloud networks.Source: §4 · p. 4

4.1.1 Diverse Mapping Ops in One Architecture.

• The Mapping Unit targets kernel mapping, k-nearest-neighbors, ball query, and farthest point sampling by converting distance comparisons into ranking operations.Ranking comparisons are processed in parallel across points.Source: §4.1 · p. 4, §4.1.1 · p. 4
• Kernel mapping becomes an intersection search between the output cloud and shifted input cloud, yielding 1.4× speedup and up to 14× area savings over hash-table mapping at equal parallelism.The merged arrays expose matches among adjacent equal coordinates.Source: §4.1.1 · p. 5

4.1.2 Mapping Unit Architecture Design.

• The sorting-network-based Mapping Unit supports the unified ranking operations without data movement between co-processors.Its comparator elements carry a comparison key and payload, while six stages perform fetching, distance calculation, sorting, buffering, merging, and intersection detection.Source: §4.1.2 · p. 6
• The sorter supports arbitrary-length Sort and TopK by iteratively merge-sorting subarrays and truncating intermediate arrays to k elements.Figure 10 also shows a forwarding loop for arbitrary-length MergeSort and logarithmic intersection detection stages.Source: §4.1.4 · p. 6, Figure 10 · p. 6
• 1.18× faster than SpAtten’s quick-selection TopK engine on average with the same parallelism, the design reuses its sorter because point-cloud k values are small relative to input size.Typical k values are 16/32/64 versus inputs of 8192.Source: §4.1.4 · p. 7

4.2.1 Data Orchestration.

• The MMU combines cache and scratchpad behavior, managing on-chip memory in tiles whose metadata can be arranged as a tag array, FIFO, or stack.Decoupled memory requests and responses allow explicit placement and replacement through the MIR Container.Source: §4.2.1 · p. 7

4.2.2 Data Flow.

• Weight-stationary inner loops reduce weight accesses, while output-stationary outer loops keep partial sums on-chip until all neighbors and input channels are traversed.The point dimension is much larger than the channel dimension.Source: §4.2.2 · p. 7

4.2.3 MMU for Sparse Computation.

• Fetch-on-demand performs matrix-vector multiplication immediately after fetching input features, saving at least 3× DRAM access for those features versus Gather-MatMul-Scatter; a direct-mapped input cache further reuses them.Cache miss rates decrease as neighbor count, channel count, and cache block size increase, but larger blocks raise DRAM miss penalty, so block sizes vary by layer.Source: §4.2.3 · pp. 7–8

4.2.4 MMU for Dense Computation.

• For dense layers, the MMU retains data on-chip until a prefetch tile conflicts, and temporally fuses consecutive FC layers by tracking layer metadata on a stack.Compilation selects tilings and drops the last layer from a fusion group when estimated intermediate-data storage would overflow for all possible tilings.Source: §4.2.4 · p. 8

4.3 Matrix Unit

• The Matrix Unit’s systolic array assigns rows of processing elements to input channels and columns to output channels, eliminating the scatter circuit by computing one output point per cycle.Source: §4.3 · p. 8

5.1 Evaluation Setup

• The eight benchmarks cover four application domains and diverse point-cloud sizes and modalities, and the hardware evaluation uses a synthesized TSMC 40nm implementation with cycle-accurate simulation.The simulator is verified against RTL and integrated with Ramulator for DRAM behavior.Source: §5.1 · p. 9

5.2.1 Speedup and Energy Savings.

• Compared with GPU products, PointAcc delivers average speedup of 3.7× and average energy savings of 22×; across GPU, TPU, and CPU products, averages are 3.7×/22×, 53×/210×, and 90×/176×, respectively.Source: §5.2.1 · p. 9

5.2.2 Comparison with Mesorasi Architecture.

• On PointNet++-based benchmarks, PointAcc.Edge averages 1.3× speedup and 11× energy savings over Mesorasi hardware; on S3DIS segmentation with Mini-MinkowskiUNet, it exceeds 100× speedup and improves mIoU by 9.1%.Mesorasi does not support independent weights for different neighbors, which are used in many point-cloud networks.Source: §5.2.2 · p. 10, Figure 15 · p. 10

5.2.3 Source of Performance Gain.

• On SemanticKITTI mapping, mergesort can be slower than CPU/GPU hash-table mapping, but PointAcc is over 10× faster than CPU/GPU through parallel comparisons and pipelined stages.On CPU/GPU, intersection detection takes nearly twice the runtime of filtering hash-table query misses because intersection processes the merged input and output clouds.Source: §5.2.3 · p. 10, Figure 17 · p. 10
• Configurable caching reduces layer DRAM access by 3.5× to 6.3×, while temporal fusion cuts DRAM access by 33% to 64% on PointNet++-based networks.For PointNet, the absence of downsampling layers allows more fusion and 1.5× to 2× more DRAM reduction than other PointNet++ networks.Source: §5.2.3 · pp. 10–11, Figure 19 · p. 11
• On MinkowskiUNet running SemanticKITTI, computation accounts for 69% of total energy and DRAM access for 23%, with matrix multiplication dominating latency.The MMU reduces DRAM access and overlaps data movement with matrix multiplication.Source: §5.2.3 · p. 11, Figure 21 · p. 11

Deep Learning on Point Clouds

• Prior point-cloud methods include voxel-based 3D CNNs, direct PointNet-family models, hybrid PVCNN, and SparseConv approaches aimed at large outdoor scenes.PointNet++ variants include Deep KD-Net, SpiderCNN, PointCNN, PointConv, and KPConv.Source: §6 · p. 11

• Figure 20 compares DRAM access size for fusion-mode memory management against independently running layers one by one.Source: Figure 20 · p. 11

Point Cloud Accelerators

• Point-cloud accelerator research has largely targeted neighbor search, with Mesorasi supporting PointNet++ networks through delayed aggregation but having limited applicability.The passage refers to Section 5.2.2 for the applicability limitation.Source: §6 · p. 11

Deep Learning Accelerators

• Conventional sparse DNN accelerators exploit unstructured zeros, but do not support modern point-cloud networks.Examples include EIE, Cambricon-X, Cnvlutin, and SCNN.Source: §6 · p. 11

• PointAcc maps newly introduced point-cloud mapping operations onto a shared ranking-based compute kernel.Source: §7 · p. 11
• It streams sparse computation with on-demand caching and temporally fuses consecutive dense layers to address the memory footprint of point-cloud sparsity.Source: §7 · p. 11
• Extensive evaluation reports speedup and energy reduction over CPU, GPU, and TPU.Source: §7 · p. 11