PaperlayerView exact revision on arXiv

Source-linked AI summary

Putting the Object Back into Video Object Segmentation

Ho Kei Cheng, Seoung Wug Oh, Brian Price, Joon-Young Lee, Alexander Schwing

arXiv:2310.12982v2cs.CV

TL;DR

Pixel-level memory matching in VOS can be noisy around distractors and weaker on challenging data. Cutie adds top-down object-level memory reading through object queries that interact iteratively with high-resolution pixel features. It reports stronger MOSE performance while maintaining or improving efficiency, but remains challenged by highly similar nearby or occluding objects.

  • Problem

    Pixel-level memory reading lacks high-level consistency and is prone to matching noise from distractors, contributing to lower performance in challenging scenarios.

  • Method

    Cutie uses trained object queries and an object transformer to combine top-down object-level representations with bottom-up high-resolution pixel features and foreground-background masked attention.

  • Results

    +8.7 J&F on MOSE over XMem is reported for Cutie, with competitive accuracy and efficiency on standard datasets.

  • Takeaways & Limitations

    Cutie provides an end-to-end VOS framework that integrates object-level memory reading with pixel-level detail for robust segmentation in challenging scenarios.

  • Takeaways & Limitations

    Cutie often fails when highly similar objects move in close proximity or occlude each other because available memory features may not be sufficiently discriminative.

Abstract

from arXiv · show

We present Cutie, a video object segmentation (VOS) network with object-level memory reading, which puts the object representation from memory back into the video object segmentation result. Recent works on VOS employ bottom-up pixel-level memory reading which struggles due to matching noise, especially in the presence of distractors, resulting in lower performance in more challenging data. In contrast, Cutie performs top-down object-level memory reading by adapting a small set of object queries. Via those, it interacts with the bottom-up pixel features iteratively with a query-based object transformer (qt, hence Cutie). The object queries act as a high-level summary of the target object, while high-resolution feature maps are retained for accurate segmentation. Together with foreground-background masked attention, Cutie cleanly separates the semantics of the foreground object from the background. On the challenging MOSE dataset, Cutie improves by 8.7 J&F over XMem with a similar running time and improves by 4.2 J&F over DeAOT while being three times faster. Code is available at: https://hkchengrex.github.io/Cutie

1. Introduction

Cutie addresses noisy pixel-level memory matching in challenging video object segmentation by adding top-down object-level reasoning. It combines object queries, high-resolution pixel features, and foreground-background separation for robust segmentation.

  • Semi-supervised VOS tracks and segments open-vocabulary objects from a first-frame annotation, supporting robotics, video editing, annotation, and universal video segmentation.
  • Pixel-level memory reading independently matches query pixels to memory pixels, but lacks high-level consistency and is vulnerable to distractors, occlusions, and matching noise.
  • More than 20 J&F points separate recent approaches’ performance on challenging MOSE from simpler DAVIS-2017, motivating object-level reasoning.
  • Cutie uses a small set of trained object queries with an object transformer to iteratively probe pixel features and encode object-level information.
  • Bidirectional communication preserves global object representations alongside high-resolution feature maps, while foreground-background masked attention separates target semantics from distractors.
  • +8.7 J&F on MOSE over XMem demonstrates improved robustness while Cutie remains competitive on DAVIS and YouTubeVOS in accuracy and efficiency.

2. Related Works

Related VOS work spans memory-based pixel matching, object-level descriptors, and automatic segmentation pipelines. Cutie targets efficiency and accuracy by combining object-level communication with high-resolution features without costly spatial-map attention.

  • Memory-based VOS stores past features for future frames, while online finetuning is slow and recurrent methods lack context under occlusion.
  • Spatial feature-map attention in prior memory methods has O(n^4) time and space complexity, whereas Cutie uses object queries for efficient global communication.
  • Cutie avoids costly spatial-map attention in its object transformer, supporting real-time operation compared with concurrent methods running below 4 frames per second.
  • Earlier object-level VOS methods offered flexible object descriptors but under-used high-resolution features, limiting segmentation accuracy on standard benchmarks.
  • Cutie differs by learning object information end-to-end, enabling bidirectional pixel-object communication in a one-stage framework without separate instance segmentation.
  • VOS also supports open-vocabulary, universal, and unsupervised video segmentation pipelines, where Cutie’s reported robustness and efficiency are intended to help.

3. Cutie

Cutie combines high-resolution pixel memory with target-specific object queries in a bidirectional object transformer. Foreground-background masking separates object and distractor semantics while positional embeddings and compact object memory support location-aware, long-term representations.

  • Overview: Cutie sequentially encodes segmented frames into high-resolution pixel memory and high-level object memory for streaming VOS.The pixel memory follows existing work, while the object memory is part of Cutie’s target-specific representation.
  • Overview: The object transformer initializes queries by summing learned static queries with dynamic object memory, then iteratively updates queries and pixel readouts through bidirectional cross-attention.Each transformer block lets queries attend to pixel features and then feeds object semantics back into the pixel features.
  • Foreground-Background Masked Attention: Foreground-background masked attention assigns half the queries to foreground and half to background, preventing cross-region attention while preserving later query communication through self-attention.The mask is derived from the current pixel features and uses 0 for allowed attention and −∞ for disallowed attention.
  • Positional Embeddings: Positional embeddings are added to queries and keys at every attention layer, combining learnable or dynamic query information with coordinate-aware pixel information.Object-query positional embeddings combine learnable embeddings with object memory; pixel embeddings combine sinusoidal coordinates with the initial readout.
  • Object Memory: The compact object memory stores N vectors summarizing the target object through mask-pooling over encoded object features.Foreground-background separation produces clean semantics during pooling, while sinusoidal positional embeddings enable location-aware aggregation.
  • Object Memory: During inference, streaming averaging makes object-memory updates constant in time and memory with respect to video length, while zero pooling weights prevent feature drifting during occlusion.

4. Experiments

Experiments compare Cutie with state-of-the-art VOS methods on standard, challenging, and long-video benchmarks, then test its design choices and limitations. Cutie is especially effective on challenging data while maintaining efficiency, and its combined top-down and bottom-up design is supported by ablations.

  • Main Results: Cutie is evaluated on DAVIS 2017, YouTubeVOS, MOSE, BURST, and LVOS using J&F, HOTA, FPS, and memory usage where applicable.MOSE tests heavy occlusions and crowded environments; BURST and LVOS test long videos.
  • Main Results: Cutie achieves better results than state-of-the-art methods, especially on the challenging MOSE dataset, while remaining efficient.The reported improvement over XMem is +8.7 J&F in MOSE, with similar running time.
  • Ablations: The dynamic object memory and static object query yield similar running times, while the reported ablation values include 67.3±0.36, 66.9±0.26, and 67.2±0.10 J&F.The ablation compares configurations with both components, without object memory, and without object query.
  • Ablations: Cutie combines high-level top-down queries with pixel-level bottom-up features for robust segmentation in challenging scenarios.The object transformer iteratively interacts with pixel features while retaining high-resolution feature maps and object-level information.
  • Ablations: Foreground-background masked attention is crucial for good performance, while full attention produces confusing signals and unstable training.The authors hypothesize that this issue is particularly important in cluttered settings.
  • Limitations: Cutie often fails when highly similar objects move in close proximity or occlude each other.The authors suspect that the pixel and object memories may lack sufficiently discriminative features in these cases.

5. Conclusion

The conclusion presents Cutie as an end-to-end VOS network using object-level memory reading for robust segmentation in challenging scenarios. It integrates top-down and bottom-up features and achieves state-of-the-art results on several benchmarks, particularly MOSE.

  • 5. Conclusion: Cutie integrates top-down and bottom-up features through object-level memory reading for robust video object segmentation.The approach is presented as an end-to-end network for challenging scenarios.
  • 5. Conclusion: Cutie achieves new state-of-the-art results on several benchmarks, especially on the challenging MOSE dataset.

Supplementary Material Putting the Object Back into Video Object Segmentation

The supplementary material presents visual comparisons and failure cases for Cutie across several video datasets. Cutie and other current methods still struggle when similar objects overlap or occlude one another.

  • Visual comparisons cover YouTubeVOS-2019 validation, DAVIS 2017 test-dev, and MOSE validation sequences.The comparisons use Cutie-base and train all models with MOSE; only first-frame ground-truth annotations are available.
  • Failure Cases: Cutie fails in challenging cases where similar objects move in close proximity or occlude each other.The supplementary material notes that current state-of-the-art methods also fail in these cases.
  • Failure Cases: All models have difficulty tracking elephants hidden behind a large unannotated foreground elephant.This failure occurs in the “elephants” sequence, including the light blue mask.
  • Failure Cases: All models fail when the pink-masked bird moves and occludes other birds.This failure occurs in the “birds” sequence.
  • Failure Cases: The authors attribute these failures to memory features that cannot disambiguate objects with similar appearance and position.They identify three-dimensional spatial understanding as a potential future direction.

C. Running Time Analysis

The supplementary analysis compares component runtime and visualizes how pixel attention and object-transformer attention differ. The visualizations show that the object transformer suppresses matching noise and improves segmentation in challenging scenes.

  • Running Time: Runtime measurements compare total component times for XMem and Cutie on one video using a 2080Ti.The authors synchronize and warm up the systems, while noting that minor implementation and runtime variations may cause small deviations.
  • Pixel attention is more spatially spread and more easily distracted by similar objects than object query attention.The object transformer uses pixel attention for initialization and object query attention to restructure features in each transformer block.
  • Feature Progression: The object transformer suppresses low-level matching noise and produces more coherent object-level features.Figure S2 visualizes these feature progressions across transformer layers.
  • Masked Attention: Masked attention helps differentiate similarly looking objects in challenging cases, although both variants work well in simpler cases.The comparison is qualitative and is shown in Figure S3.
  • Object Transformer: Using the object transformer leads to more complete and accurate outputs compared with omitting object queries.Figure S4 compares the model without object queries, the default Cutie model, and ground truth.

E. Details on BURST Evaluation

The BURST evaluation modifies memory updates and image resolution to accommodate long videos and high-resolution inputs. Additional long-term memory and inference-time settings provide supported avenues for performance improvement.

  • BURST Evaluation: BURST updates memory every 10th frame and downsizes images so the shorter edge is at most 600 pixels.These settings follow prior work and replace the default 480-pixel shorter edge for all methods.
  • BURST Evaluation: BURST performance is assessed with HOTA separately for common and uncommon object classes.The evaluation follows the BURST protocol.
  • Long-Term Memory: A plug-in long-term memory supplements Cutie’s default FIFO memory by compressing working memory instead of discarding it.The long-term memory can be adopted without retraining, using XMem’s default parameters.
  • Cutie+: Cutie+ increases maximum memory frames from 5 to 10, reduces the memory interval from 5 to 3, and raises inference resolution from 480 to 720 pixels.These hyperparameter changes require no retraining but increase running time; results are reported in Table S2.

F.2. Comparisons with Methods that Use External Training

The supplementary material compares Cutie with methods using external training data and introduces a larger-data MEGA training scheme. Some comparison details rely on estimates or results reported in Table S2.

  • External Training: Table S2 compares Cutie with SimVOS, JointFormer, and ISVOS, whose methods use external training.The authors could not obtain code for these methods; ISVOS running time is estimated because it does not report one.
  • MEGA Training: The MEGA scheme adds BURST and OVIS to DAVIS, YouTubeVOS, and MOSE, with 50K additional training iterations.The corresponding results are reported in the bottom portion of Table S2.

F.3. Results on YouTubeVOS-2018 and LVOS

Cutie is evaluated on YouTubeVOS-2018, LVOS, and multi-object inference settings, with particular attention to long-term videos, tiny objects, speed, memory, and seed variation.

  • LVOS results: LVOS has 50 validation videos and 50 test videos, whereas BURST has 988 validation videos and 1419 test videos.Cutie was tested on LVOS after its design was completed, using long-term memory and no dataset-specific tuning.
  • LVOS results: Cutie-base outperforms LVOS’s DDMemory baseline on the test set, matches it approximately on validation, and runs about twice as fast.LVOS contains substantially more tiny objects than other evaluated datasets, creating a distinct challenge for methods not designed for small-object detection.
  • Performance variations: Across five random seeds, Cutie-small’s gains over XMem correspond to +24.2 s.d. on MOSE and +8.2 s.d. on YouTubeVOS-2019.The reported comparisons are 8.7 J&F on MOSE and 0.9 G on YouTubeVOS over XMem.
  • Efficiency: Cutie-small remains real-time at 29.9 FPS with five objects, while BURST averages 5.57 object tracks and DAVIS-2017 averages 2.03.Processing speed decreases as the number of objects increases, but remains real-time for common object counts.
  • Memory usage: The default FIFO memory bank provides constant GPU memory usage over time, while optional long-term memory uses less memory than XMem because Cutie uses 256 channels versus XMem’s 512.DeAOT’s memory usage is described as unbounded in the comparison.

G.3.2 Main Training

Cutie’s main training combines mixed video sampling, stable augmentation, point-supervised losses, and an efficient decoder and memory design derived partly from prior work.

  • Main training: Main training mixes DAVIS-2017 and YouTubeVOS-2019, optionally adding MOSE, while sampling eight-frame sequences with a curriculum for temporal distance.DAVIS is sampled twice as often because its annotations are considered more accurate.
  • Main training: Training applies shared geometric augmentation across sequence frames, including mirroring, affine transformation, cut-and-paste, resized crops, color jittering, and grayscaling.The same crop and rotation are applied to all frames in a sequence.
  • Losses: Point supervision samples uncertain and uniform points for loss computation, reducing loss-computation memory while producing no significant segmentation-accuracy difference versus XMem’s loss.The loss combines equally weighted cross-entropy and soft dice terms.
  • Decoder: Cutie uses a 128-channel decoder instead of XMem’s 256-channel decoder, improving running time without an observed performance drop.The authors attribute this to input features refined by the object transformer.
  • Memory: Its pixel memory design derives the attentional and recurrent components from XMem, and XMem’s long-term memory can be added for long-video evaluation without retraining.The recurrent component updates a hidden state with a GRU, while deep updates refresh that state during memory insertion.

H. Interactive Tool for Video Segmentation

Cutie is extended into an interactive video segmentation tool that combines user annotations with Cutie’s propagation through video.

  • Interactive workflow: Users annotate one or more frames with an interactive image segmentation tool, then use Cutie to propagate those segmentations through the video.The tool follows MiVOS’s decoupled paradigm.
  • Interactive workflow: Permanent memory frames can be added to increase segmentation robustness during interactive use.This follows the approach used in XMem++.
  • Availability: The interactive tool is open-source for researchers, data annotators, and video editors.
Loading 2310.12982v2…