PaperlayerView exact revision on arXiv

Source-linked AI summary

SpaceCache: Intermediate Feature Reuse for Efficient Semantic Segmentation on Point Cloud Streams

Dongheon Yang, Beomseok Kang, Jae-Joon Kim

arXiv:2610.21207v1cs.CV

TL;DR

Point-cloud streams require repeated inference over large token sets, while streaming acceleration has received relatively limited attention. SpaceCache reuses cached intermediate features for previously observed regions, with cache management to keep reuse reliable. Across outdoor and indoor benchmarks, it improves the accuracy-efficiency trade-off over token-merging methods, reaching up to 1.65× speedup within a 1-point mIoU drop.

  • Problem

    Point-cloud segmentation repeatedly processes large token sets in streams, but accelerating streaming point-cloud perception remains relatively underexplored.

    Source: §1 · p. 1

  • Method

    SpaceCache skips transformer computation for previously observed regions by reusing their cached intermediate features, with confidence- and lifetime-based cache management.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    Across outdoor and indoor benchmarks with Utonia and LitePT-B, SpaceCache reaches up to 1.65× speedup within a 1-point mIoU drop and consistently outperforms token-merging methods.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Across semantic-segmentation benchmarks, SpaceCache consistently improves the accuracy-efficiency trade-off over existing token-reduction methods with low runtime and memory overhead.

    Source: §6 · p. 10

    Reliable reuse requires overlapping-region features from previous frames to preserve consistent semantic information in the current frame.

    Source: §2.2 · p. 4

Abstract

from arXiv · show

Point-cloud streams are central to autonomous driving and robotics, yet existing segmentation networks process every frame from scratch, although consecutive frames largely overlap. We observe that the intermediate features of point-cloud transformers remain semantically consistent across frames for the same spatial region, especially in deep stages. Building on this, we present SpaceCache, a training-free feature cache that reuses these features across frames. Tokens of previously observed regions skip a sequence of transformer blocks and take their cached features, only the remaining tokens are computed, and the full token layout is restored for dense prediction. Lightweight cache management keeps reused features reliable by limiting each entry's age and evicting low-confidence entries. Unlike token merging, SpaceCache selects tokens at negligible cost and runs its cache operations once per encoder stage. On both outdoor and indoor benchmarks, such as nuScenes, Waymo, and ScanNet, with two transformer backbones of Utonia and LitePT-B, SpaceCache achieves up to 1.65x speedup with less than 1-point mIoU drop and consistently outperforms token-merging methods.

1 INTRODUCTION

Point-cloud segmentation must process very large token sets at every timestep, while streaming inference acceleration remains relatively underexplored. SpaceCache reuses intermediate features for previously observed regions, motivated by greater overlap and semantic consistency at deeper stages, and is reported to improve the accuracy-efficiency trade-off over token-merging methods.

1 INTRODUCTION

  • Point-cloud streams repeatedly process tens to hundreds of thousands of tokens per frame, while accelerating streaming point-cloud perception remains relatively underexplored.The cited motivation identifies low-latency inference as critical for autonomous vehicles and robots.Source: §1 · p. 1
  • Token-merging methods reduce computation but incur similarity-estimation and irregular aggregation overhead; temporal point-cloud methods also may lose context or require learned compensation.The introduction contrasts these costs with SpaceCache's training-free reuse of intermediate features.Source: §1 · pp. 1–2
  • Deeper encoder stages show greater cross-frame token overlap and feature consistency, motivating reuse of features from previously observed regions.Figure 2 describes both overlap and Reuse Agreement as increasing with depth.Source: §1 · p. 2, Figure 2 · p. 3
  • SpaceCache addresses stale or unreliable cached features by combining confidence-based commit and eviction with lifetime-based eviction.The stated risks include moving objects and ambiguous regions.Source: §1 · p. 2
  • Across nuScenes, Waymo, and ScanNet with Utonia and LitePT-B, SpaceCache reaches up to 1.65× speedup within a 1-point mIoU drop and consistently outperforms token-merging methods.The reported comparison covers both outdoor and indoor benchmarks.Source: §1 · p. 2

2 BACKGROUND AND PRELIMINARY STUDY

Point-cloud encoders keep token coordinates fixed within each stage, allowing spatial correspondences to persist across its blocks. The preliminary study finds that deeper stages have greater temporal overlap and Reuse Agreement, motivating deeper-feature reuse.

2.1 NETWORK PROPERTY

  • Within a stage, transformer blocks preserve token coordinates, so cross-frame spatial matches remain valid across all blocks.Each token is a point or spatially aggregated representation with a 3D coordinate and a feature vector.Source: §2.1 · p. 3

2.2 FEATURE REUSE OPPORTUNITY

  • Overlap rises with depth from 4% to 49% on nuScenes, 13% to 66% on Waymo, and 64% to 88% on ScanNet.Progressive downsampling makes deeper tokens represent larger spatial regions, increasing the likelihood that the same regions recur across consecutive frames.Source: §2.2 · p. 4
  • Reuse Agreement rises from approximately 75–83% at Stage 0 to 91–93% at Stages 3–4, beginning to saturate from Stage 3.The measure tracks whether reused features and their corresponding current-frame features are associated with the same semantic category.Source: §2.2 · p. 4
  • Deeper stages combine higher temporal overlap with stronger semantic consistency, making their intermediate representations increasingly suitable for temporal feature reuse.The study examines both how many tokens overlap across frames and whether their features remain semantically consistent.Source: §2.2 · pp. 3–4

3 METHOD

SpaceCache reuses cached intermediate features for previously observed regions while computing only cache misses, then restores the full token layout. Confidence filtering and age limits manage cache reliability, while applying reuse at deeper stages reduces computation with modest overhead.

3.1 OVERVIEW OF SPACECACHE

  • Cache-hit tokens bypass selected stage blocks; the Reuse Gate combines cached and newly computed features and restores the full token layout for later layers.The Commit Gate updates the cache after prediction by retaining valid entries and candidates.Source: §3.1 · p. 4, Figure 4 · p. 5

3.2 OPERATION DETAILS OF SPACECACHE

  • Current-frame coordinates are transformed to global coordinates and quantized into stage-dependent spatial cells so observations of the same region receive the same index.SpaceCache looks up region indices with parallel GPU binary searches; newly computed cache-miss features become candidates for later validation.Source: §3.2 · p. 5

3.3 CACHE MANAGEMENT IN SPACECACHE

  • Confidence filtering discards low-confidence candidates and evicts low-confidence cached entries; a maximum lifetime also expires entries as reuse agreement declines with age.New entries receive expiration ages from 1 to L, staggered by a hash of the region index; a smaller L refreshes features more often but adds computation.Source: §3.3 · p. 6, Figure 5 · p. 6

3.4 COMPUTATION REDUCTION AND OVERHEAD

  • Cache hits reduce the tokens processed by encoder blocks, and SpaceCache primarily targets Stages 3 and 4, which have high temporal overlap and feature reusability.Cache operations take about 5 ms per frame and use approximately 70 MiB with single-precision features; their costs are better amortized as more blocks are bypassed.Source: §3.4 · p. 6

4 EXPERIMENTAL RESULTS

Across outdoor and indoor comparisons, SpaceCache is faster than token-merging baselines at the reported accuracy-drop settings. Ablations examine cache reliability and stage selection, while the efficiency analysis attributes gains to broader computation skipping and low cache-operation overhead.

4.1 COMPARISON WITH PRIOR METHODS

  • SpaceCache is fastest in all four outdoor settings, reaching 1.65× on Waymo with LitePT-B (307→186 ms) while removing 49% of FLOPs.The comparison uses each method’s fewest-FLOPs setting within a 1-point mIoU drop.Source: §4.1 · p. 8
  • On ScanNet with LitePT-B, SpaceCacheStage-3 is fastest at both reported mIoU-drop settings, reaching 1.22× with 36% fewer GFLOPs within the 2-point drop.Kernel time is averaged over 50 representative frames; the reported speedup is smaller than outdoors because denser indoor points are merged more aggressively by downsampling.Source: §4.1 · p. 8, Table 2 · p. 8

4.2 ABLATION STUDY

  • With confidence filtering, increasing cache lifetime from 1 to 32 raises speedup to 1.37× but lowers mIoU by 0.65 points; without filtering, mIoU falls 4.81 points at lifetime 32.At similar speedups, filtering yields 66.91% mIoU at 1.25× for lifetime 8 versus 65.84% at 1.24× for lifetime 4 without filtering.Source: §4.2 · p. 8
  • Applying SpaceCache to stages 3 and 4 gives the default balance of larger speedup and small accuracy drop; speedup peaks at stage 3, which contains 12 of 24 blocks.Accuracy loss shrinks from stage 1 to stage 4, consistent with greater semantic consistency at depth.Source: §4.2 · p. 8

4.3 EFFICIENCY ANALYSIS

  • On Waymo with Utonia, SpaceCache removes 37% of total FLOPs and turns that into a 25% latency reduction, while GitMerge3D’s repeated scoring and gather/reduce add 136 ms overhead.SpaceCache cache operations run once per stage and cost 4.8 ms; ToMe removes 23% of FLOPs but reduces latency by only 7%.Source: §4.3 · p. 9, Figure 7 · p. 9
  • SpaceCache raises peak memory by 95 MiB (7.0%), comprising 69 MiB for the cache and 26 MiB of activations for cache-miss features.The activations are held until the Commit Gate.Source: §4.3 · p. 9, Figure 7 · p. 9

5 RELATED WORK

Prior point-cloud acceleration methods reduce tokens, simplify computation, or reuse information across frames, but often involve training or runtime overhead. SpaceCache instead reuses deep-stage features from existing pretrained backbones without training.

Efficient point-cloud backbones

  • Point-cloud backbones have been accelerated through random sampling, superpoints, or simplified attention, but these designs require training, which is costly for pretrained-at-scale backbones.SpaceCache instead targets existing pretrained backbones on streams without training.Source: §5 · p. 9

Token reduction in vision

  • Vision token reduction includes merging, pruning, and halting, while point networks also group points; these approaches generally require training except for merging.Merging can repeat similarity searches at every block and incur slow memory accesses when accumulating tokens.Source: §5 · p. 9

Temporal reuse in point-cloud streams

  • Few temporal point-cloud methods save computation: TempNet propagates key-frame features with trained modules, SSCATeR recomputes changed-point convolutions, and EVE prunes inputs and trains a module to recover lost context.SpaceCache keeps all input points, reuses only deep-stage features, and needs no training.Source: §5 · pp. 9–10

6 CONCLUSION

The conclusion presents SpaceCache as a training-free way to reuse deep-stage features in point-cloud streams, improving the accuracy-efficiency trade-off with low runtime and memory overhead. The appendix defines token regions through the downsampling chain and describes how token accuracy is weighted by covered points.

6 CONCLUSION

  • Across outdoor and indoor semantic-segmentation benchmarks, SpaceCache improves the accuracy-efficiency trade-off over token-reduction methods with low runtime and memory overhead.The method reuses deep-stage features while preserving dense token representations.Source: §6 · p. 10

A.1 TOKEN REGIONS

  • A pooling chain records each downsampling layer’s mapping from tokens to the next stage, allowing each stage token to represent a spatial region of input points.Each input point reaches exactly one token at a given stage.Source: §A.1 · p. 14
  • Token accuracy scores the covered input points, weighting each token by its covered-point count and averaging frames by their covered-point counts.Points with the ignore label are excluded.Source: §A.1 · p. 14

A.2 REUSE AGREEMENT

Reuse Agreement compares the class-associated distributions of a current feature and its reused counterpart. Its value is one for identical distributions and zero when they share no class.

A.2 REUSE AGREEMENT

  • Reuse Agreement measures whether a reused feature predicts the same class as the current-frame feature for the same region.Class prototypes are means of current-frame features from pure tokens, and classes need at least 20 such tokens to receive a prototype.Source: §A.2 · p. 14
  • Feature-class distributions come from softmaxed cosine similarities to class prototypes, with stage temperatures calibrated to equalize mean normalized entropy.Agreement is the overlap between the current and reused feature distributions: one for identical distributions and zero for none in common.Source: §A.2 · p. 14
  • For Figure 2(c), agreement averages over overlapped tokens in 60 consecutive frame pairs per dataset, then averages across three datasets for each backbone.For Figure 5, the reused feature is a cached or candidate feature.Source: §A.2 · p. 14

B IMPLEMENTATION DETAILS

The implementation applies shared pretrained weights without further training and configures SpaceCache and three token-merging baselines. SpaceCache uses per-stage spatial-region caches, while ToMe and ALGM merge tokens at specified encoder stages.

B IMPLEMENTATION DETAILS

  • All methods use the same pretrained weights without training, and are implemented in Pointcept; batch size is 1 and attention uses dense attention.SpaceCache is compatible with flash-attention, but GitMerge3D's attention-score bias is not.Source: B IMPLEMENTATION DETAILS · p. 15

B.1 SPACECACHE

  • Each SpaceCache stage uses its own voxel-sized cache, averages features for tokens in the same region, and clears the cache at each sequence start.Region lookups, feature assembly, and cache updates use fused Triton kernels on RTX A6000 or PyTorch on Jetson AGX Orin.Source: §B.1 · p. 15
  • SpaceCache packs shifted region coordinates into a 64-bit key using 24 bits each for two axes and 15 bits for the third.The encoding is one-to-one within the stated coordinate ranges, and its range doubles at each stage.Source: §B.1 · p. 15
  • Cache entries expire on a region-specific fixed L-frame cadence, refreshing roughly one L-th of resident entries per frame.A lookup counts an entry as a hit only before its due frame.Source: §B.1 · p. 15

B.2 TOME

  • ToMe matches tokens across alternating sets within each attention patch using key cosine similarity, then averages selected pairs until decoder unpooling restores their features.Its sweep parameter r is the fraction of tokens merged per patch in each block.Source: §B.2 · p. 16

B.3 ALGM

  • ALGM performs local similarity-threshold merging at stage 0 and global bipartite matching at stage 3, with both merges reversed by decoder copying.The global merge retains pairs above a threshold based on the mean and standard deviation of best-match similarities.Source: §B.3 · p. 16

B.4 GITMERGE3D

GitMerge3D merges tokens within attention patches before attention and copies the resulting features back immediately. The implementation also reports a non-adaptive version and sweeps each method's main parameter.

B.4 GITMERGE3D

  • GitMerge3D selects random destination tokens within bins and averages a fraction r of patch tokens into their most similar destinations using value-vector cosine similarity.Merging occurs inside every attention block, and outputs are copied back to the original tokens immediately.Source: §B.4 · p. 16
  • At the same r=0.5, adaptive patch partitioning lowers Waymo accuracy considerably but preserves accuracy on nuScenes and ScanNet, while its overhead offsets savings on the latter two.The paper reports GitMerge3D without adaptive partitioning in its tables.Source: §B.4 · p. 16, Table 4 · p. 16

B.5 SWEPT SETTINGS

  • The sweep varies r for ToMe and GitMerge3D, τ for ALGM's local merge, and θ for SpaceCache; SpaceCache uses L=8 outdoors and L=4 indoors.Table 5 lists swept values and marks the selected point within a 1.0-point mIoU drop.Source: §B.5 · p. 17, Table 5 · p. 17

C DATASET AND EXPERIMENTAL SETUP

The ScanNet video stream is constructed from sampled RGB-D validation scenes with projected points and benchmark labels. Accuracy uses full validation sets, while efficiency metrics use representative streamed sequences and platform-specific latency measurements.

C.1 SCANNET VIDEO STREAM

  • The ScanNet stream keeps every tenth frame from 10 validation scenes, totaling 2,221 frames, and back-projects depth pixels closer than 8 m with their nearest color pixels.Frames without valid camera poses are dropped.Source: §C.1 · p. 18
  • ScanNet point labels come from the corresponding 2D label images mapped to 20 benchmark classes, and all other labels are ignored.Frames use scene coordinates derived from camera poses so the sequence shares one voxel grid; LitePT-B points take normals from nearest mesh vertices.Source: §C.1 · p. 18

C.2 EFFICIENCY MEASUREMENT

  • Accuracy uses full validation sets, but latency, FLOPs, and memory use representative sequences selected to match full-set voxel-count distributions.Latency is measured on RTX A6000 for outdoor scenes and Jetson AGX Orin for indoor scenes; ScanNet FLOPs cover all 10 scenes.Source: §C.2 · p. 18
  • FLOPs count linear layers, sparse convolutions, attention, and merging similarities, treating each multiply-accumulate as two FLOPs.Normalizations, activations, softmax, and gathers are excluded.Source: §C.2 · p. 18

C.3 METRICS OF THE ABLATION STUDY

The ablation metrics compare cache-hit fractions and point-accuracy changes across static and dynamic regions. The breakdown attributes computation savings and overhead to scoring, feature movement, and how often each operation runs.

C.3 METRICS OF THE ABLATION STUDY

  • The ablation reports cache-hit fraction and point-accuracy change separately for dynamic regions whose movable-class label changed and static regions whose label stayed unchanged.The accuracy-change baseline is the backbone at L=1.Source: §C.3 · p. 18

D.1 FLOPS SAVING AND SCORING OVERHEAD

  • Table 6 reports that SpaceCache removes the most network-layer FLOPs and adds the least scoring FLOPs among the compared methods.The comparison is made at each method's operating point within a 2.0-point mIoU drop on representative Waymo scenes with Utonia.Source: §D.1 · p. 19, D BREAKDOWN DETAILS · p. 19
  • SpaceCache scoring costs O(N_l) once per stage because it rotates coordinates, whereas merging methods compare token features and incur method-specific scoring costs.GitMerge3D's scoring is O(WDN_l) per block; ToMe's is O(WDN_l/H) per block.Source: §D.1 · p. 19

D.2 LATENCY OF THE OVERHEAD OPERATIONS

  • SpaceCache gathers each cache-miss token once without accumulation, then writes computed misses and cached hits to original positions; merging baselines accumulate source features into destinations.Its sorted cache is updated once per stage and frame, evicting expired and low-confidence entries.Source: §D.2 · p. 19
  • SpaceCache runs its operations once per stage, while GitMerge3D repeats scoring and feature movement every block; ToMe repeats scoring and gathering every block.ALGM performs operations twice, at stages 0 and 3; overhead latency also reflects memory traffic, not just FLOPs.Source: §D.2 · p. 19

D.3 MEMORY OVERHEAD

SpaceCache’s memory overhead comes from cached features and, for cache-miss tokens, activations held until the Commit Gate. Its cache uses 69.4 MiB across stages three and four, while memory overhead varies across comparison methods.

D.3 MEMORY OVERHEAD

  • 69.4 MiB is the combined cache memory for SpaceCache’s third and fourth stages.Stage three uses 47.7 MiB and stage four uses 21.7 MiB; each entry stores a region key, feature, and age.Source: §D.3 · p. 20
  • 18 MiB and 26 MiB of added activations are reported for SpaceCacheStage-3 and SpaceCacheStage-3,4, respectively.These activations hold newly computed cache-miss features until the Commit Gate after the decoder.Source: §D.3 · p. 20
  • GitMerge3D stays within 2 MiB of the backbone’s activation memory, while ToMe and ALGM add 175 MiB and 58 MiB, respectively.GitMerge3D restores tokens after each attention and retains nothing across blocks; ToMe and ALGM retain merge maps and unmerged tokens until decoder restoration.Source: §D.3 · p. 20
  • Table 6 reports FLOPs, latency, and memory breakdowns for representative scenes at each method’s point within a 2.0-point mIoU drop.FLOPs and latency are per frame and use dense attention; memory uses flash-attention.Source: Table 6 · p. 20

Related papers

Loading 2610.21207v1…