PaperlayerView exact revision on arXiv

Source-linked AI summary

RS$^3$-Prune: Read-Sparse, Store-Sparse Token Pruning for Video Object Segmentation

Avilasha Mandal, Sarvesh Shashikumar

arXiv:2608.22526v1cs.CV

TL;DR

Memory-bank VOS must manage growing token budgets to run on long videos, multiple streams, and memory-bounded GPUs. RS^3-Prune prunes tokens on the read and write sides; across benchmarks, it yields +38.8% FPS and −13.1% VRAM overall without retraining.

  • Problem

    Memory-bank VOS faces the challenge of scaling to long videos, many simultaneous streams, and memory-bounded GPUs without exhausting memory or stalling per frame.

    Source: §1 · p. 1

  • Method

    RS^3-Prune applies training-free read- and write-side token pruning at the encoder-to-attention and memory-encoder-to-bank boundaries, leaving the backbone, memory module, and decoder unchanged.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    +38.8% FPS and −13.1% VRAM overall, with no retraining or change to released checkpoints.

    Source: §6 · p. 14

  • Takeaways & Limitations

    Across benchmark datasets, RS3-Prune retains more accuracy than other efficient-VOS methods at lower time and memory budgets.

    Source: §8 · p. 14

    For objects traveling far outside its dilated seed-frame footprint, the fixed write keep-set trades FPS for J&F.

    Source: §7 · p. 14

Abstract

from arXiv · show

We introduce RS$^3$-Prune, a training-free token-pruning recipe that instantiates as a small set of inference time hooks atop existing video object segmentation (VOS) networks. Modern VOS models have converged on a common, expensive design: an image encoder produces a dense token grid for every frame, and a memory bank accumulates these tokens across all previously processed frames to condition future predictions. As a video grows longer, the resulting token budget governs both per-frame latency and peak GPU memory. Hence these models break on use cases such as --- long-form video or real-time deployment on memory-bounded accelerators. In this work we argue that the right axis along which to compress memory-bank VOS is the token budget itself. RS$^3$-Prune operates in two precise locations within an arbitrary memory-bank VOS pipeline: at the boundary between the image encoder and the memory-attention readout, where we restrict the queries that participate in the cross-frame attention to only a small, geometrically informed subset; and at the boundary between the memory encoder and the memory bank, where we restrict which tokens are ever permitted to enter the bank to those that lie within the object's spatial extent. Over various established benchmarks, RS$^3$-Prune delivers up to $38.8\%$ FPS speedup and reduces $13.1\%$ peak memory usage, while preserving a competitive $\mathcal{J}$&$\mathcal{F}$ compared to the unmodified VOS networks.

1 Introduction

RS^3-Prune targets the token budget that limits memory-bank VOS deployment, pruning both current-frame reads and long-term writes without training. The paper also motivates its geometric prior and evaluates the recipe across five architectures and five benchmarks.

Introduction

  • VOS is near benchmark saturation, but scaling to long videos, simultaneous streams, and memory-bounded GPUs is limited by token flow through the memory pipeline.The introduction frames deployment scale—not benchmark segmentation capability—as the pressing challenge for current systems.Source: §1 · p. 1
  • Dense per-frame token grids accumulate as memory key/value pairs, making cross-frame attention cost and peak memory grow with frame resolution and video length.The memory bank conditions subsequent predictions, so throughput falls and parallel workloads can exhaust memory as it grows.Source: §1 · pp. 1–2
  • RS^3-Prune applies training-free read-side pruning to current-frame queries and write-side pruning to tokens committed to the memory bank.The two gates sit between the image encoder and memory-attention readout, and between the memory encoder and memory bank, respectively.Source: §1 · p. 2
  • The paper mathematically justifies its geometric prior using encoder-token L2 energy and evaluates RS^3-Prune on five published architectures across DAVIS-17 [18], MOSE [8], SA-V [20], YouTube-VOS [24], and LVOSv2 [9] validation sets.The benchmarks are described as deliberately different in character; the paper says it reports both strengths and limits.Source: §1 · p. 2

2 Related Work

Related work establishes memory-bank cross-attention as a shared VOS bottleneck and shows that prior token-reduction methods focus on attention-time queries, whereas RS^3-Prune also targets memory writes.

Image and video object segmentation

  • SAM introduced promptable image segmentation, while SAM2 [20] extended it to video with streaming memory, at a larger token budget than purpose-built VOS architectures.SAM2 stores frame predictions in a fixed-size rolling buffer and reads them to condition segmentation of the next frame.Source: §2 · p. 2

Memory bank video object segmentation

  • Online fine-tuning [2] [10] incurred high per-video computation, propagation methods traded robustness for speed, and Space-Time Memory Networks [16] established growing-bank readout.XMem [6], Cutie [7], and SAM2 [20] refine this blueprint, but retain its dense-token, growing-memory cross-attention bottleneck targeted by RS^3-Prune.Source: §2 · pp. 2–3

Token pruning

  • Prior pruning, merging, and token-transformation methods [19] [13] [25] [23] [27] [4] [14] reduce tokens at the attention-time query boundary; RS^3-Prune also targets memory writes.The paper identifies memory-write pruning as a structurally distinct opportunity for VOS and argues that operating at both boundaries enables its intended deployment regimes.Source: §2 · p. 3

3 Method

RS3-Prune adds read- and write-side token pruning to the encoder–attention–memory-bank–decoder pipeline, leaving the pretrained backbone, memory module, and decoder unchanged. It uses no retraining, fine-tuning, or new parameters.

3.1 Read-side pruning: a per-frame, per-object spatial gate

  • Read-side pruning limits memory-attention queries to high-norm encoder tokens inside dilated object priors, propagated from the model’s previous-frame predictions after initialization.When an object prediction is empty, a cached bounding box expands with the missing-frame streak, subject to a cap, to support re-entry recovery.Source: §3.1 · p. 6

3.2 Write-side pruning: which tokens enter the bank

  • Write-side pruning admits only tokens in a dilated seed-mask region, using a fixed token count per frame to reduce memory-bank growth.The seed mask is downsampled to the encoder grid and dilated; the same keep-set is used for subsequent writes. If its footprint covers over 95% or under 5% of the grid, the original write path is used.Source: §3.2 · pp. 7–8

3.3 Per-frame morphological closure

  • A per-object 9 × 9 morphological closure fills small interior holes in predicted masks before the final per-pixel argmax.The closure is applied at decoder output resolution and does not grow the object’s outer boundary.Source: §3.3 · p. 8

3.4 Why the geometric prior works: token energy on the encoder grid

  • The geometric prior selects object-region tokens whose encoder norms tend to be largest, while dilation accommodates uncertainty in previous-frame predictions.The passages argue that this spatial filter refines feature-only energy ranking by avoiding a threshold on a noisy scalar.Source: §3.1 · p. 6, §3.4 · p. 9

3.5 Cross-Architecture Portability

  • RS3-Prune applies to memory-bank VOS networks that expose both a query-based memory-attention readout and a token insertion path into long-term storage.Source: §3.5 · p. 9

4 Experiments

RS3-Prune applies read- and write-side token pruning to memory-bank VOS, achieving average speed and memory gains while keeping J &F competitive. Its geometric prior also distinguishes it from image-domain token-reduction methods that struggle in video settings.

4.1 Datasets and Implementation Details

  • The evaluation covers DAVIS-17, SA-V, YouTube-VOS, MOSE, and LVOSv2, whose validation clips average ∼1.14 minutes and stress-test long-video deployment.All scores use a single NVIDIA RTX-4090-class GPU with batch size 1; RS3-Prune uses the published checkpoints, read-side ρ=0.3, and network-specific write-side dilation without retraining.Source: §4.1 · p. 9, §4.2 · p. 9

4.2 Main result across five benchmarks

  • 38.8% mean FPS speedup and 13.1% mean peak VRAM reduction are achieved across architectures and datasets with measured timings versus unmodified networks.Averaged over five benchmarks (four for STM), the gains are +58% FPS and −32% VRAM on STM, +26% FPS and −16% VRAM on XMem, +60% FPS and −7% VRAM on Cutie, +28% FPS and −4% VRAM on SAM2, and +22% FPS and −6% VRAM on SAM3.1.Source: Table 1 · p. 10, §4.2 · p. 10
  • At ρ=0.3, SAM2’s memory-attention query budget falls from 4096 tokens to ≈1228, or 3.3× fewer, while J &F improves on several sequences.The geometrically supported foreground tokens discard background context that can act as noise; the passage reports lifts on SAM2 SA-V, YT-VOS, and MOSE while running faster and lighter.Source: §4.2 · pp. 10–11
  • The reported gains and competitive J &F come from pruning alone, without human-in-the-loop prompts between frames; the same efficiency profile carries over when prompting is added to both pipelines.Masks are propagated from the seed annotation under each backbone’s standard VOS protocol, with prompting treated equally for pruned and vanilla networks.Source: §4.2 · p. 11

4.3 Comparison of RS3-Prune against prior efficient-VOS methods

  • At ρ=0.30, SparseViT, DynamicViT [19], and VLTP [4] incur large accuracy losses in vanilla VOS, while RS3-Prune’s geometric prior avoids dropping tracked-object tokens.The comparison concerns image-domain token-reduction methods without an object- or video-level prior, which can retain background and discard foreground tokens in dynamic video settings.Source: §4.3 · p. 11, Table 2 · p. 11

Ablation

The ablation isolates the read-side and write-side prunes, the read-side keep ratio ρ, and the write-side dilation Dw. It examines their effects across architectures and benchmarks.

5.1 Read-side vs. write-side across architectures

  • On XMem and Cutie, adding write-side pruning yields ~25% further FPS gains and 6%/2.5% VRAM reductions over read-only pruning at negligible J&F cost.Read-side pruning supplies essentially all throughput gains, while write-side pruning chiefly reduces VRAM. On SAM2, write-side pruning trades 1.5 J&F points for +12% FPS; on STM, it improves J&F by 3.5 points and reduces VRAM by 30.2% relative to Vanilla.Source: Table 3 · p. 12, §5.1 · p. 12

5.2 Keep ratio ρ within the spatial prior

  • Across ρ∈{0.2,0.3,0.5,0.7}, XMem DAVIS-17 J&F stays within 0.846–0.853, while FPS drops 1.5× and VRAM grows 10% as ρ increases.The geometric prior confines queries to relevant cells, and ranking them by visual energy makes J&F-relevant positions converge quickly; admitting more tokens adds background energy without changing J&F. The authors adopt ρ=0.3 for efficiency at near-optimal J&F.Source: §5.2 · pp. 12–13, Table 4 · p. 13

5.3 Write-side dilation on SAM2

  • On SAM2, smaller Dw shrinks the bank for more FPS at the cost of J&F, while larger Dw retains more of the bank and trades FPS back; Dw=24 is the default.The Dw=24 configuration is used in the main results reported in Table 1.Source: Table 5 · p. 13, §5.3 · p. 13

6 Discussion

RS3-Prune’s read-side and write-side token pruning target different resources, combining throughput and memory savings while withholding background context that can act as noise.

  • The read-side prune caps queries reaching memory attention to drive throughput gains, while the write-side prune shrinks the stored bank to deliver most VRAM reduction.Their effects add because they act on different resources. Averaged over five benchmarks, the reported pairs are STM +58% FPS/−32% VRAM, XMem +26% FPS/−16% VRAM, Cutie +60% FPS/−7% VRAM, SAM2 +28% FPS/−4% VRAM, and SAM3.1 +22% FPS/−6% VRAM.Source: §6 · pp. 13–14
  • Across five benchmarks, RS3-Prune yields +38.8% FPS and −13.1% VRAM with no retraining or changes to released checkpoints.Source: §6 · p. 14
  • Geometrically supported foreground tokens can improve accuracy by withholding background context that full-softmax readout would otherwise integrate as noise.The passage points to the green numbers in table 1 as evidence.Source: §6 · p. 14
  • Unlike query-only pruning [4] [19], which cannot shrink Nmem, write-side pruning compresses Nmem and peak memory in the long-video regime.The two pruning operations are complementary.Source: §6 · p. 14

7 Limitation

RS3-Prune’s fixed write keep-set can miss objects that travel far from the seed-frame footprint or require costly dilation to cover them. Its read keep-set can also depend on prior predictions, though a fallback helps after catastrophic prediction loss.

7 Limitation

  • A fixed write keep-set based on the dilated seed-frame footprint can under-cover far-traveling objects, harming J &F, or require wider dilation that costs FPS.The paper identifies a learned, motion-history-aware write keep-set as a plug-in extension.Source: §7 · p. 14
  • Because the write keep-set stays fixed in memory across object occurrences, one object moving completely out of frame will not disrupt masks for other objects.Source: §7 · p. 14
  • The read keep-set propagates from the previous prediction, and after catastrophic prediction loss the streak-counted bounding-box fallback helps, though re-detection could shorten recovery.The paper describes a dedicated re-detection branch as a plug-in extension rather than a redesign.Source: §7 · p. 14

8 Conclusion

RS3-Prune makes the token budget in memory-bank VOS tunable through a training-free geometric recipe that prunes tokens at the read and write boundaries. Across benchmark datasets, it retains more accuracy than other efficient-VOS methods at lower time and memory budgets.

  • Across benchmark datasets, RS3-Prune retains far more accuracy than other efficient-VOS methods at lower time and memory budgets.Source: §8 · p. 14
  • RS3-Prune uses a training-free geometric recipe to restrict tokens at the image-encoder-to-memory-attention read boundary and the memory-encoder-to-memory-bank write boundary.The read side is the speed lever, and the write side is the memory lever; the network’s consolidation machinery remains intact.Source: §8 · p. 14
  • The recipe applies to any memory-bank VOS network that exposes the two encoder boundaries, making its token budget a tunable knob rather than an architectural constant.Source: §8 · p. 14

Appendix · Algorithm at a glance

RS^3-Prune is a training-free, parameter-free inference-time pipeline that reuses frozen-network tensors and applies deterministic geometric operations to sparsify memory reads and writes. Its keep-set sizes govern per-frame attention cost and memory-bank growth, while fixed write sets and read/write containment preserve compatibility with the backbone’s memory logic.

Algorithm at a glance

  • The pipeline requires no learned parameters or backbone retraining, reusing frozen-network tensors and deterministic geometric operations.The listed operations include dilation, downsampling, and top-k selection.Source: Algorithm at a glance · p. 17
  • Read attention uses at most ρhw queries, scaling read-side cost by ρ, while the bank stores O(T·|Kw|) entries instead of O(T·hw).The keep-sets are recomputed with cheap geometric operations whose combined cost is negligible next to one memory-attention layer.Source: Algorithm at a glance · p. 17
  • Kw is fixed once at τo and reused for every write in the sequence, preserving the bank’s constant-per-frame-stride consolidation logic.Source: Algorithm at a glance · pp. 17–18
  • When Dw ≥ rs + c, P(t) ⊆ Kw at every frame, so reads never query positions that were not written.Source: Algorithm at a glance · p. 18
  • For a text prompt, the initial object box is rasterized from a grounded detector, and SAM 2’s memory bank is seeded through its native prompt path.Source: Algorithm at a glance · p. 18
  • For each object, the predicted mask is downsampled and closed with a 9×9 operation; the algorithm then selects Zt[Kw] and adds its memory encoding to the bank.Source: Algorithm at a glance · p. 18

Related papers

Loading 2608.22526v1…