PaperlayerView exact revision on arXiv

Source-linked AI summary

XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Model

Ho Kei Cheng, Alexander G. Schwing

arXiv:2207.07115v2cs.CV

TL;DR

Long-video VOS methods face a trade-off between memory consumption and segmentation quality. XMem addresses it with three connected memory stores and memory potentiation that consolidates actively used working-memory elements into compact long-term storage. It greatly exceeds prior state-of-the-art results on long videos while remaining on par with state of the art on short videos.

  • Problem

    Long-video VOS links memory consumption to segmentation quality because existing methods either store many representations or compress high-resolution features early.

  • Method

    XMem combines sensory, high-resolution working, and compact long-term feature memories with memory consolidation, potentiation, and a unified reading operation.

  • Results

    XMem greatly exceeds prior state-of-the-art results on Long-time Video and is on par with current state-of-the-art methods on short-video datasets.

  • Takeaways & Limitations

    The multi-store design supports accurate VOS on long and short videos with minimal GPU memory usage and is presented as a step toward accessible VOS on mobile devices.

  • Takeaways & Limitations

    XMem sometimes fails when target objects move too quickly or exhibit severe motion blur because its sensory memory cannot update fast enough.

Abstract

from arXiv · show

We present XMem, a video object segmentation architecture for long videos with unified feature memory stores inspired by the Atkinson-Shiffrin memory model. Prior work on video object segmentation typically only uses one type of feature memory. For videos longer than a minute, a single feature memory model tightly links memory consumption and accuracy. In contrast, following the Atkinson-Shiffrin model, we develop an architecture that incorporates multiple independent yet deeply-connected feature memory stores: a rapidly updated sensory memory, a high-resolution working memory, and a compact thus sustained long-term memory. Crucially, we develop a memory potentiation algorithm that routinely consolidates actively used working memory elements into the long-term memory, which avoids memory explosion and minimizes performance decay for long-term prediction. Combined with a new memory reading mechanism, XMem greatly exceeds state-of-the-art performance on long-video datasets while being on par with state-of-the-art methods (that do not work on long videos) on short-video datasets. Code is available at https://hkchengrex.github.io/XMem

1 Introduction

XMem addresses the trade-off between memory consumption and segmentation quality in long-video VOS by combining three temporal-scale memory stores. Its design targets accurate, efficient, real-time, online segmentation across long and short videos.

  • Motivation: Long-video VOS methods often require substantial GPU memory and struggle beyond one minute, while memory-efficient methods sacrifice segmentation quality through early feature compression.Attention-based methods store many past-frame representations; long-video methods merge high-resolution features immediately.
  • XMem: XMem uses rapidly updated sensory memory, high-resolution working memory, and compact sustained long-term memory as independent but deeply connected stores.The stores operate at different temporal scales to separate short-term smoothness, high-resolution historical context, and sustained compact storage.
  • XMem: Its memory reading operation combines features from all three stores to produce high-quality masks while keeping GPU memory usage low.The unified reading design is intended to handle both long and short videos.
  • Results: XMem greatly exceeds prior state-of-the-art results on Long-time Video and matches current state-of-the-art performance on short-video datasets.The short-video comparison is against methods that cannot handle long videos.
  • XMem: XMem consolidates representative working-memory prototypes and enriches them with memory potentiation for compact long-term storage.The approach is designed to preserve useful information while controlling memory growth.

2 Related Works

Prior VOS methods use feature memories, online adaptation, recurrence, or increasingly large historical banks, each with limitations for speed, occlusion handling, context, memory, or accuracy. XMem instead uses multiple memory stores to avoid eager compression while supporting long-term prediction.

  • Prior VOS methods: Online-learning VOS methods adapt network weights at test time, which typically slows inference.Recent improvements make adaptation more efficient but remain sensitive to input.
  • Prior VOS methods: Tracking-based and recurrent methods are efficient but often lack long-term context, drift, or lose track after occlusions.Including an initial reference frame helps global matching but provides limited context as videos progress.
  • Feature-memory methods: Recent attention-based methods store more past frames as feature memory, improving context but increasing GPU memory requirements for long videos.The related-work discussion identifies STM and its extensions as influential examples of this feature-memory strategy.
  • XMem: XMem’s multi-store feature memory avoids eagerly compressing new high-resolution features and achieves higher accuracy in short-term and long-term predictions.This contrasts with methods that merge new features into a compact representation immediately.

3 XMem

XMem combines sensory, working, and long-term feature memories with attention-based reading to segment query frames. It controls memory growth by consolidating representative, frequently used working-memory features into a compact long-term store.

  • Overview: XMem initializes multiple feature stores and reads from long-term, working, and sensory memory to generate masks for subsequent query frames.The decoder combines memory-read features with sensory memory, while the architecture supports multiple target objects.
  • Memory Reading: An affinity-controlled readout maps each query element across all long-term and working-memory elements and aggregates their values into feature F.The affinity matrix is obtained by applying softmax over memory elements to key-query similarities.
  • Memory Reading: XMem’s anisotropic L2 similarity adds key-associated shrinkage and query-associated selection terms to encode confidence and query-specific channel importance.The shrinkage term controls local influence through confidence, while the selection term emphasizes more discriminative key-space channels.
  • Memory Consolidation: When working memory reaches Tmax, XMem retains the first and most recent Tmin −1 frames while converting the remaining Tmax −Tmin frames into long-term representations.Prototype keys are selected from candidate features, and prototype values are enriched through non-local aggregation before appending both to long-term memory.
  • Working Memory: The working memory appends a query key and an image-and-mask value every r-th frame, while limiting stored frames to avoid memory explosion.Its size is bounded by Tmin ≤T < Tmax, with excess frames consolidated into long-term memory.

4 Experiments

Experiments show that XMem scales to long videos while retaining strong short-video performance, using multi-store memory and consolidation to control memory growth. Ablations support the roles of its memory stores, reading design, consolidation, and sensory updates, while failures remain under severe motion blur or very fast motion.

  • Long-video evaluation: XMem uses at most 1.4GB of GPU memory by capping long-term memory at 10,000 elements.The default setting yields a 6328% compression ratio from working to long-term memory.
  • Long-video evaluation: Long-video methods trade off memory, speed, and accuracy: local windows lose context, while growing memory banks require sparse insertion and scale poorly.These behaviors are documented for CFBI(+), JOINT, STM, AOT, and STCN.
  • Benchmark results: XMem scales to longer videos without the degradation seen in competing approaches and remains strong in short-term performance.AFB-URR scales without degradation but has lower short-term performance because it compresses features eagerly.
  • Ablations: Removing long-term memory slightly improves resolution-dependent performance but prevents long-video handling and slows inference.Removing working memory leaves sensory memory with constant memory cost, but long-term memory cannot function.
  • Ablations: Memory consolidation is the most effective tested strategy, while deep sensory updates improve performance without noticeable speed loss at every r-th frame.More frequent deep updates add value-encoder calls and slow processing.
  • Limitations: XMem sometimes fails when target objects move too quickly or exhibit severe motion blur because sensory memory cannot catch up.The authors suggest a larger-receptive-field sensory memory as a possible improvement.

5 Conclusion

The paper concludes that XMem is, to the authors’ knowledge, the first multi-store feature-memory model for video object segmentation. It reports strong performance with low GPU memory usage on both long and short videos and points toward more accessible VOS.

  • XMem is presented as the first multi-store feature memory model used for video object segmentation, to the authors’ best knowledge.
  • The work aims to support accessible video object segmentation on mobile devices and draw attention to long-term VOS.
  • The appendix covers memory consolidation, qualitative comparisons, failure cases, processing-rate scaling, retrained-baseline results, dataset variation, multi-scale evaluation, and efficient similarity implementation.

A Visualizing Memory Consolidation

The visualization illustrates how XMem consolidates candidate working-memory frames into compact long-term prototypes. Aggregation weights focus on semantically meaningful regions across objects and scenes.

  • Candidate frames are converted into prototype representations through aggregation weights mapped to prototype positions.The weights aggregate candidate values into prototype values.
  • The examples show consolidation focusing on meaningful regions including a swan’s beak, vegetation, riverbank, torso, legs, arms, trees, and wall.

B Qualitative Results

Qualitative results show XMem maintaining effective segmentation across long, fast-moving, and heavily occluded sequences where competing methods lose context, compress features, or miss rapid changes. Its main observed failure mode is very fast motion with blur.

  • Dressage: On the 10,767-frame dressage sequence, JOINT loses track, AFB-URR is less accurate, and STCN misses fast changes because of sparse memory insertion.
  • Breakdance: On breakdance, XMem performs well despite a single foreground object undergoing large and fast motion.The sequence contains 18,187 frames.
  • Cans: In the cans sequence, the two cans are occluded for more than 2,000 frames, creating a long-term reappearance challenge.The cans have similar appearances and the sequence contains five objects.
  • Failure cases: XMem fails on some birds with similar appearances and on objects undergoing large motion blur or rapid motion.
  • Scaling: Reducing LTmax saves memory and increases speed, while XMem’s FPS plateaus after reaching the long-term-memory limit.The default LTmax of 10,000 can store information from around 3,900 frames after consolidation.
  • Scaling: STCN slows as its memory bank expands, whereas XMem maintains relatively stable and fast FPS through memory consolidation.

E Re-training STCN

The re-trained STCN baseline shows that the altered training setup is not sufficient by itself to explain improved performance.

  • On average, the re-trained STCN shows no significant performance difference from the original STCN.It achieves 1.2 higher J &F on DAVIS and 1.0 higher G on YouTubeVOS, but the average change is insignificant.

F Results on YouTubeVOS 2019 validation

This section reports quantitative comparisons on YouTubeVOS 2019 validation and measures FPS using the 2018 version.

  • The YouTubeVOS 2019 validation results are tabulated, with FPS measured on the 2018 version.The two versions’ FPS values are highly correlated because their average object counts and video lengths are similar.

G Results with Different Training Datasets

The paper evaluates XMem under different training-data configurations and compares it with methods trained without static-image pretraining.

  • The method is pretrained on static images using a mixture of single-object datasets before video-dataset training.The paper also reports results when training on DAVIS 2017 only, YouTubeVOS 2019 only, or both datasets.
  • Table S4 compares methods that do not use static-image pretraining.
  • Tables S5–S9 report XMem performance across DAVIS and YouTubeVOS datasets under different training-data settings.These include DAVIS 2016, DAVIS 2017 validation and test-dev, and YouTubeVOS 2018 and 2019 validation.

H Multi-scale Evaluation

Multi-scale evaluation combines independently processed augmented inputs by averaging their output probability maps, with dataset-specific scale choices.

  • Multi-scale Evaluation: Multi-scale evaluation averages output probability maps from independently processed augmented inputs.The evaluation uses scale changes or vertical mirroring, with P = 512 for relaxed compression and vertical mirroring enabled.
  • DAVIS: On DAVIS, a single 720p scale performs better than merging multiple smaller scales.The method uses r = 3, and the test-dev set additionally includes r = 5 as a multi-temporal-scale merge.
  • YouTubeVOS: On YouTubeVOS, the adopted scales are {480, 528, 576, 624}, while larger scales are unhelpful.Multiple temporal scales are not used on YouTubeVOS.

I Implementation of the Anisotropic L2 Similarity

The implementation decomposes anisotropic L2 similarity into tensor operations for efficient computation. It uses shrinkage and selection terms, with broadcasting and element-wise operations enabling a fully vectorized implementation.

  • The implementation decomposes anisotropic L2 similarity into a sequence of tensor operations for memory- and compute-efficient execution.The decomposition is motivated by the need for a practical implementation of the proposed similarity function.
  • Given keys k, values v, queries q, shrinkage terms s, and selection terms e, similarity is computed between each key element and query element.Keys and queries have channel-spatial dimensions, while shrinkage is associated with key elements and selection with query positions.
  • Setting si = ecj = 1 for all i, j, and c recovers the original L2 similarity.
  • The implementation uses element-wise operations, matrix multiplications, and broadcasting, with “:” denoting all elements, “@” singleton broadcast dimensions, and ⊙ Hadamard products.The all-ones row vector 1 has length Ck.
Loading 2207.07115v2…