PaperlayerView exact revision on arXiv

Source-linked AI summary

RS$^3$-Prune: Read-Sparse, Store-Sparse Token Pruning for Video Object Segmentation

Avilasha Mandal, Sarvesh Shashikumar

arXiv:2608.22526v1cs.CV

TL;DR

Modern memory-bank VOS systems process dense token grids and accumulate memory tokens, making long-video and memory-bounded deployment costly. RS3-Prune is a training-free recipe that prunes read queries and stored tokens at two pipeline boundaries. Across benchmarks, it reports 38.8% FPS speedup and 13.1% peak VRAM reduction while preserving competitive J & F.

  • Problem

    Dense per-frame tokens and accumulated memory-bank tokens make VOS increasingly costly for long videos and memory-bounded deployment.

  • Method

    RS3-Prune applies training-free read-side and write-side token pruning to existing memory-bank VOS networks without changing their backbones, memory modules, or decoders.

  • Results

    38.8% mean FPS speedup and 13.1% mean peak VRAM reduction are reported relative to unmodified networks across measured architectures and datasets.

  • Takeaways & Limitations

    RS3-Prune compresses the token budget while retaining competitive J & F across benchmarked VOS networks.

  • Takeaways & Limitations

    A fixed write keep-set can under-cover objects that travel far outside the seed-frame footprint, trading either FPS or J & F.

Abstract

from arXiv · show

We introduce RS$^3$-Prune, a training-free token-pruning recipe that instantiates as a small set of inference time hooks atop existing video object segmentation (VOS) networks. Modern VOS models have converged on a common, expensive design: an image encoder produces a dense token grid for every frame, and a memory bank accumulates these tokens across all previously processed frames to condition future predictions. As a video grows longer, the resulting token budget governs both per-frame latency and peak GPU memory. Hence these models break on use cases such as --- long-form video or real-time deployment on memory-bounded accelerators. In this work we argue that the right axis along which to compress memory-bank VOS is the token budget itself. RS$^3$-Prune operates in two precise locations within an arbitrary memory-bank VOS pipeline: at the boundary between the image encoder and the memory-attention readout, where we restrict the queries that participate in the cross-frame attention to only a small, geometrically informed subset; and at the boundary between the memory encoder and the memory bank, where we restrict which tokens are ever permitted to enter the bank to those that lie within the object's spatial extent. Over various established benchmarks, RS$^3$-Prune delivers up to $38.8\%$ FPS speedup and reduces $13.1\%$ peak memory usage, while preserving a competitive $\mathcal{J}$&$\mathcal{F}$ compared to the unmodified VOS networks.

1 Attentive AI

The passage identifies the Indian Institute of Technology Delhi.

  • Indian Institute of Technology Delhi is identified in the passage.
  • The institution is presented without additional research context.
  • No method, result, or benchmark information appears in this passage.

1 Introduction

Modern memory-bank VOS systems face deployment limits because dense tokens are processed and accumulated across video frames. RS3-Prune addresses this bottleneck by pruning tokens at both read and write boundaries.

  • VOS deployment bottlenecks arise from the number of tokens flowing through the memory pipeline, rather than mainly from algorithmic capability.
  • Modern memory-bank networks encode dense per-frame token grids and accumulate key/value representations in a bank for later predictions.
  • RS3-Prune reduces the token budget at the image-encoder/readout boundary and the memory-encoder/bank boundary.
  • The method is presented as a training-free, symmetric recipe evaluated across five published architectures and five deliberately varied benchmarks.

2 Related Work

Prior VOS systems evolved from online fine-tuning and propagation toward memory-bank architectures, while token-pruning research developed selective dropping and token merging for dense vision.

  • SAM introduced prompt-driven image segmentation, and SAM2 extended the paradigm to video with streaming memory.
  • Early VOS methods used online fine-tuning, which provided strong accuracy but incurred prohibitive per-video computation.
  • Propagation-based methods accelerated inference by warping masks, trading some robustness for speed.
  • Token-pruning methods selectively drop or merge vision tokens using attention, energy, latency, or similarity-based strategies.

3 Method

RS3-Prune adds two training-free token gates to memory-bank VOS: one sparsifies current-frame queries before memory attention, and the other limits tokens written to long-term memory. The method uses object-derived spatial priors, token energy, temporal propagation, and disappearance recovery while leaving pretrained network components unchanged.

  • 3 Method: RS3-Prune inserts read-side and write-side pruning at the image-encoder/memory-attention and memory-encoder/memory-bank boundaries.The read gate controls current-frame queries, while the write gate controls long-term memory admission.
  • 3.1 Read-side pruning: The read-side keep-set intersects a dilated object spatial prior with the highest-energy encoder tokens.Bounding boxes are projected onto the token grid, dilated, unioned, and filtered by token norms.
  • 3.1 Read-side pruning: From the second frame onward, the spatial gate follows the model’s previous predictions without an additional supervisory signal.The first-frame mask seeds the prior; subsequent priors are rebuilt from predicted per-object masks.
  • 3.1 Read-side pruning: A streak-counted fallback progressively dilates the last non-empty bounding box when an object disappears, up to a capped radius.The streak resets when a non-empty mask returns, while the cached box persists throughout the video.
  • 3.2 Write-side pruning: The write-side keep-set selects seed-anchored spatial cells before memory encoding, preventing background clutter from accumulating in the bank.Per-object seed masks are unioned and pooled to the token grid; the retained region includes a motion-related margin proportional to Dw.
  • 3.4 Why the geometric prior works: The geometric prior supplies object-versus-background support, while energy ranking supplies textured-versus-smooth support for retaining high-energy tokens.The combined set is expected to retain tokens with the largest energy and discard tokens with near-zero energy.
  • 3.5 Cross-Architecture Portability: RS3-Prune applies to memory-bank VOS networks exposing both a memory-attention readout and a memory-bank insertion path.The backbone, memory module, and mask decoder remain unchanged, and the method runs on the original pretrained checkpoint without retraining.

4 Experiments

RS3-Prune is evaluated across five VOS benchmarks and five memory-bank architectures without retraining, targeting efficiency while maintaining competitive segmentation accuracy. It improves throughput and reduces peak VRAM, with gains attributed solely to read- and write-side pruning.

  • Evaluation setup: RS3-Prune is evaluated on five standard VOS benchmarks using five established memory-bank architectures.The benchmarks are DAVIS-17, SA-V, YouTube-VOS, MOSE, and LVOSv2; architectures include STM, XMem, Cutie, SAM2, and SAM3.1.
  • Evaluation setup: The comparison uses each architecture’s published checkpoint versus the full RS3-Prune configuration, with no retraining.The full configuration uses read-side ρ=0.3 and network-specific write-side dilation.
  • Main results: 38.8% mean FPS speedup and 13.1% mean peak VRAM reduction are achieved relative to unmodified networks.The averages span architectures and datasets with measured timings in Table 1.
  • Main results: J &F remains competitive across datasets and architectures, and improves over vanilla on several sequences despite using fewer query tokens and less context.The retained tokens are geometrically supported foreground tokens, while discarded background context can otherwise act as noise.
  • Evaluation conditions: RS3-Prune’s efficiency gains are attributable to token pruning alone, without human-in-the-loop prompts during inference.Masks are propagated from the seed annotation according to each backbone’s standard VOS protocol.
  • Comparison: Image-domain token-reduction baselines incur large accuracy losses when applied to vanilla VOS, whereas RS3-Prune uses a geometric object prior.The cited comparison includes SparseViT, DynamicViT, and VLTP.

Ablation

The ablations separate RS3-Prune’s speed and memory mechanisms and examine the read keep ratio and write dilation. Read pruning primarily drives throughput, write pruning primarily reduces VRAM, and parameter choices expose accuracy–efficiency tradeoffs.

  • Ablation design: The ablation isolates read-side pruning, write-side pruning, read keep ratio ρ, and write-side dilation Dw.Table 3 compares Vanilla, read-side only, and the full recipe across architectures.
  • Read-side versus write-side: On XMem and Cutie, adding write pruning yields about 25% further FPS gain and additional VRAM reduction at negligible J &F cost.The further VRAM reductions are 6% on XMem and 2.5% on Cutie relative to read-side-only rows.
  • Read-side versus write-side: On SAM2, write pruning trades 1.5 J &F points for 12% higher FPS, while on STM it improves J &F by 3.5 points and reduces VRAM by 30.2% versus Vanilla.These architecture-specific effects show that write pruning can act as an accuracy–efficiency knob.
  • Read-side versus write-side: The read-side prune is the speed lever, while the write-side prune is the memory lever.Read pruning caps per-frame memory-attention queries; write pruning shrinks the stored memory bank.
  • Read keep ratio: Across ρ values from 0.2 to 0.7, XMem’s J &F stays within 0.7 points while FPS drops 1.5× and VRAM grows 10% as ρ increases.The paper adopts ρ=0.3 as the efficiency point with near-optimal J &F.
  • Write-side dilation: Larger Dw retains more stored positions and trades FPS for J &F; Dw=24 is used as the default configuration.The sweep is performed on SAM2 with DAVIS-17 validation data.

6 Discussion

RS3-Prune separates token pruning into a read-side speed lever and a write-side memory lever, with complementary benefits and a structural explanation for possible accuracy gains.

  • The read-side prune caps memory-attention queries per frame and carries essentially all throughput gain.
  • The write-side prune shrinks the stored bank and delivers most of the VRAM reduction.
  • +58% FPS/−32% VRAM on STM, +26% FPS/−16% VRAM on XMem, +60% FPS/−7% VRAM on Cutie, and +28% FPS/−4% VRAM on SAM2 were averaged over five benchmarks.
  • +22% FPS/−6% VRAM on SAM3.1 contributed to an overall +38.8% FPS and −13.1% VRAM, with no retraining or checkpoint changes.
  • Keeping geometrically supported foreground tokens withholds background context that full-softmax readout would otherwise integrate as noise.
  • Unlike query-only pruning, the write-side prune compresses Nmem and therefore peak memory, making the two prunes complementary.

7 Limitation

The fixed write keep-set creates a trajectory-dependent trade-off, while read-side recovery remains exposed to catastrophic prediction loss.

  • The write keep-set is fixed at the seed-frame footprint dilated by Dw throughout the video.
  • Objects traveling far outside that footprint require dilation that either costs FPS or undercovers trajectories at the cost of J &F.
  • Under catastrophic prediction loss, the bounding-box fallback carries the system, but a dedicated re-detection branch could shorten recovery.

8 Conclusion

RS3-Prune treats unbounded token-budget growth as the deployment ceiling in memory-bank VOS and compresses it at both read and write boundaries with a training-free geometric recipe.

  • The deployment ceiling is unbounded token-budget growth through memory pipelines rather than the algorithmic gap between strongest networks.
  • RS3-Prune compresses tokens between the image encoder and memory attention for speed and between the memory encoder and bank for memory.
  • The recipe runs atop any VOS network, leaves consolidation machinery intact, and requires no training changes.
  • Across benchmark datasets, RS3-Prune retains more accuracy than other efficient-VOS methods at lower time and memory budgets.
  • The recipe makes the token budget a tunable knob for memory-bank VOS networks exposing the two identified encoder boundaries.

Algorithm at a glance

Algorithm 1 applies deterministic read and write keep-sets to a frozen VOS pipeline, reducing attention queries and stored memory tokens while preserving backbone memory logic.

  • Algorithm 1 reuses frozen-network tensors and applies deterministic dilation, downsampling, and top-k operations without learned parameters or backbone retraining.
  • The read side attends |K_r^(t)| ≤ ρ hw queries instead of hw, scaling read attention cost by ρ relative to dense processing.
  • The write side stores |K_w| ≪ hw tokens per frame, reducing bank growth from O(T · hw) to O(T · |K_w|).
  • K_w is fixed once at τ_o and reused for every write, so the bank’s constant-stride consolidation and pruning logic remains unchanged.
  • When D_w ≥ r_s + c, P(t) ⊆ K_w holds, ensuring read queries target positions that were written to memory.
  • The pipeline seeds the memory bank from the initial mask, derives a dilated write mask, and emits sparse memory tokens.
Loading 2608.22526v1…