PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking Space-Time Networks with Improved Memory Coverage for Efficient Video Object Segmentation

Ho Kei Cheng, Yu-Wing Tai, Chi-Keung Tang

arXiv:2106.05210v2cs.CV

TL;DR

Video object segmentation needs efficient and robust space-time matching, but conventional object-specific memory and dot-product affinities underuse memory. STCN uses direct frame correspondences with diversified L2-based voting, achieving state-of-the-art results while running at 20+ FPS for multiple objects. Its simplicity and efficiency support its use as a baseline, although global matching can confuse far-apart objects with similar appearance.

  • Problem

    Conventional dot-product affinities can let a small fixed subset of memory nodes dominate votes, underusing the memory bank regardless of the query.

  • Method

    STCN constructs target-agnostic correspondences directly between frames and replaces dot-product affinity with negative squared Euclidean distance for diversified memory aggregation.

  • Results

    STCN achieves new state-of-the-art results on DAVIS and YouTubeVOS while running at 20+ FPS for multiple objects.

  • Takeaways & Limitations

    Direct image-to-image correspondence and diversified voting provide a simple, efficient, and robust framework for video object segmentation.

  • Takeaways & Limitations

    Because STCN uses only global matching without temporal consistency, it may incorrectly segment far-away objects with similar appearance.

Abstract

from arXiv · show

This paper presents a simple yet effective approach to modeling space-time correspondences in the context of video object segmentation. Unlike most existing approaches, we establish correspondences directly between frames without re-encoding the mask features for every object, leading to a highly efficient and robust framework. With the correspondences, every node in the current query frame is inferred by aggregating features from the past in an associative fashion. We cast the aggregation process as a voting problem and find that the existing inner-product affinity leads to poor use of memory with a small (fixed) subset of memory nodes dominating the votes, regardless of the query. In light of this phenomenon, we propose using the negative squared Euclidean distance instead to compute the affinities. We validated that every memory node now has a chance to contribute, and experimentally showed that such diversified voting is beneficial to both memory efficiency and inference accuracy. The synergy of correspondence networks and diversified voting works exceedingly well, achieves new state-of-the-art results on both DAVIS and YouTubeVOS datasets while running significantly faster at 20+ FPS for multiple objects without bells and whistles.

1 Introduction

The paper introduces STCN, which uses direct, target-agnostic image-to-image correspondences and diversified affinity voting for efficient video object segmentation. This combination improves memory utilization and achieves strong accuracy and speed.

  • 1 Introduction: STCN propagates features from the first frame through intermediate features stored as memory, supporting semi-supervised segmentation across video frames.The task begins with a given first-frame segmentation and infers masks for subsequent frames.
  • 1 Introduction: STCN is presented as a minimalistic alternative to STM that is simpler, more efficient, and more effective without additional complications.The paper positions the method as a strong baseline for future video object segmentation work.
  • 1 Introduction: STCN builds a single RGB-based affinity matrix between frames, allowing every target object to reuse the same correspondences.This avoids constructing object-specific memory banks and affinities, making matching more efficient and robust.
  • 1 Introduction: The method replaces dot-product affinity with L2 similarity so every memory node can contribute instead of a fixed subset dominating votes.Dot products encode feature magnitude as confidence, suppressing some memory nodes regardless of the query.
  • 1 Introduction: The resulting framework achieves new state-of-the-art performance on DAVIS and YouTubeVOS while running at 20+ FPS for multiple objects.The authors attribute the result to the synergy between correspondence modeling and diversified voting.

2 Related Works

The related-work discussion connects STCN to correspondence learning, few-shot matching, and space-time memory networks. It distinguishes STCN by deliberately using image-to-image correspondences even when input masks are available.

  • Correspondence Learning: Correspondence learning is fundamental in computer vision and supports applications including optical flow and object tracking.Prior approaches learn local or deep correspondences for matching visual elements.
  • Correspondence Learning: The paper frames video object segmentation as a few-shot matching problem in which the memory bank serves as the support set.Queries are compared with many memory points using similarity functions such as cosine similarity or squared Euclidean distance.
  • Video Object Segmentation: Space-time memory networks construct an object-specific memory bank and propagate segmentation through repeated memory readout.Many subsequent methods modify STM's training, augmentation, memory readout, optical flow, or memory growth.
  • Video Object Segmentation: STCN argues that building correspondences between images is preferable to treating mask-free matching as merely a concession when masks are available.Its affinity construction is redefined to operate between frames only, alongside changes to the similarity function.

3 Space-Time Correspondence Networks (STCN)

STCN computes mask-independent frame correspondences with shared key features, then transfers memory values through a shared affinity matrix for efficient multi-object segmentation. Its memory management avoids temporary last-frame memory and reduces repeated computation compared with STM.

  • 3.1 Feature Extraction: STCN uses a key encoder on images and a value encoder on image-mask pairs, making key features independent of masks and reusable across frames.The query key can later serve as a memory key without extra computation.
  • 3.1 Feature Extraction: A single RGB-derived affinity matrix is shared across target objects instead of constructing object-specific affinities.Only memory-value reading must be repeated for multiple objects; affinity computation is reused.
  • 3.2 Memory Reading and Decoding: The memory-reading pipeline forms normalized affinities between memory keys and query keys, aggregates memory values into vQ, and passes the result to the decoder for mask generation.The aggregated readout is computed as vQ = vMW.
  • 3.3 Memory Management: STCN stores each memory frame's key and value, reusing the query key while computing the object-specific value after mask generation.Memory values depend on both the image and object mask, unlike memory keys.
  • 3.3 Memory Management: STCN omits STM's temporary last-frame memory because shared key encoders provide robust matching and avoid overly similar keys that can cause drifting.This change also reduces value-encoder calls and contributes to speedup.
  • 3.3 Memory Management: For a video of length L with m objects and memory size T < L, STCN computes affinities L times and invokes the value encoder mT times, versus STM's mL affinity computations.The comparison establishes STCN's computational advantage under the stated memory configuration.

4 Computing Affinity

STCN constructs affinities between memory and query keys, then replaces dot-product voting with negative squared Euclidean distance to improve memory coverage and efficiency. Experiments show that L2 diversifies contributions and performs well with a reduced key space and small computational overhead.

  • Affinity construction: The affinity matrix contains pairwise similarities between memory key positions and query key positions, computed using a chosen similarity function.Memory keys represent past frames, while query keys represent the current frame.
  • Dot-product limitation: Dot-product affinity makes large-magnitude memory keys dominate voting independently of query features, suppressing other memory nodes.This underutilizes the memory bank and reduces effective diversity and robustness.
  • L2 similarity: L2 similarity avoids dot-product suppression by assigning each query a nearest-region structure in which every memory point can contribute.Figure 2 illustrates this contrast using a Voronoi-style partition for L2 similarity.
  • L2 similarity: Softmax contributions under L2 change smoothly across queries, whereas dot product can assign persistently low weights to some points; cosine similarity shares the benefit but needs temperature tuning.L2 works without extra temperature tuning in the reported experiments.
  • Experimental verification: Under L2, only 0.06% of memory nodes never exceed 1% weight, compared with around 3% under dot product.The fraction contributing less than 10% at best is 7% with L2 versus 31% with dot product.
  • Efficiency: L2 similarity benefits from a reduced key-space dimension, improving memory efficiency and running time while adding only small computational overhead.The paper reports that L2 works best with a reduced key space.

5 Implementation Details

Training uses temporally ordered video frames with curriculum-controlled temporal distances, while inference memorizes every fifth frame without temporary frames. Top-k memory filtering is adopted with k = 20.

  • Training: Models are pretrained on static images with synthetic deformation and then trained on YouTubeVOS and DAVIS using Adam and PyTorch.Training uses two 11GB 2080Ti GPUs, with batch sizes of 16 for pretraining and 8 during main training.
  • Training: Each training sample contains three temporally ordered frames, using predicted frames as additional memory and gradually increasing temporal distance from 5 to 25.The schedule uses curriculum learning before annealing.
  • Memory reading: Top-k filtering is used for memory-read augmentation because it consistently improves running time, with k = 20 adopted for STCN.Kernelized memory reading is slower and does not always help in the reported experiments.
  • Inference: During inference, STCN memorizes every fifth frame and uses no temporary frame.The comparison uses full floating-point precision on a 2080Ti GPU.

6 Experiments

Experiments evaluate STCN on DAVIS and YouTubeVOS using standard segmentation metrics, speed measurements, and qualitative visualizations. The method achieves strong benchmark performance while reducing component runtime and maintaining direct, mask-free correspondences.

  • Evaluation setup: Experiments primarily use DAVIS 2017 and YouTubeVOS 2018 validation sets, with additional results on DAVIS 2016 and YouTubeVOS 2019.The study also includes DAVIS 2017 test-dev results in supplementary material.
  • Evaluation setup: DAVIS evaluation reports J, F, and J&F, while YouTubeVOS reports seen and unseen J and F scores plus the overall G score.Multi-object FPS is computed over the complete DAVIS 2017 validation set.
  • Visualizations: The paper reports mask-free correspondences that associate every pixel, including background bystanders, and directly propagate labels across frames.The visualization includes an occlusion case where the affinity lacks a distinct peak.
  • Benchmark results: STCN’s base model reaches 84.2 G on YouTubeVOS 2019, exceeding the previous challenge winner’s 82.0 G.With ensemble and multi-scale testing, the submitted model reached 86.7 G and ranked first at submission time.
  • Running time: STCN’s affinity matching is faster because it computes one raw-image affinity rather than one affinity per object, while its value encoder is lighter than STM’s memory encoder.The runtime comparison controls for top-k filtering and uses Ck = 64 in both implementations.

7 Limitations

STCN’s simple global matching omits temporal consistency, limiting its ability to distinguish distant objects with similar appearance.

  • Without local matching or optical flow, STCN may incorrectly segment far-away objects with similar appearance.The paper shows this limitation in a failure case and suggests adding temporal consistency in future extensions.

8 Conclusion

The paper concludes that STCN combines direct image-to-image correspondence with L2 similarity into a simple, efficient, and effective VOS framework. It is presented as a potential baseline backbone for future work, with broader impacts that include both beneficial and malicious uses.

  • STCN uses direct image-to-image correspondence for efficient, robust matching and L2 similarity based on an affinity analysis.
  • STCN is described as a simple, effective, and efficient framework for video object segmentation.
  • The authors hope STCN can serve as a new baseline backbone for future contributions.
  • VOS software can support video editing and safer self-driving cars, but malicious use may enable mass surveillance or privacy-infringing tracking.

B Additional results on memory scheduling

Additional experiments examine memory scheduling, benchmark settings, and implementation choices, showing interactions among similarity functions, temporal memory, spatial cues, and evaluation enhancements.

  • Additional results on memory scheduling: STM experiences a significant performance drop without the temporary frame, supporting the need for close-range relations in its robust matching.
  • Additional results on memory scheduling: 31% FPS improvement accompanies removal of the temporary frame, while STCN with dot-product similarity suffers only a slight performance drop.This supports the reported robustness of STCN’s key features under reduced memory scheduling.
  • Additional results on memory scheduling: STCN with L2 similarity performs best without the temporary frame, although the result suggests greater susceptibility to drifting under careless memory scheduling.The authors still conclude that L2 similarity is beneficial overall.
  • DAVIS test-dev: Encoding memory every third frame instead of every fifth raises DAVIS test-dev performance above state of the art, with added computation and memory costs.The efficient STCN model remains runnable on the same 11GB GPU and is reported as the fastest among competitors.

F.3.1 Pre-training

Static-image pretraining uses multiple salient-object datasets and synthetic video transformations to prepare the model for video object segmentation.

  • Pretraining uses ECSSD, FSS1000, HRSOD, BIG, and both DUTS training and testing sets.BIG and HRSOD images are downsized to a 512-pixel longer edge and sampled five times more often because of higher-quality annotations.
  • Each training sample contains three synthetic frames augmented from the same image during static-image pretraining.
  • Augmentations include random scaling, flips, color jitter, grayscale conversion, affine transforms, rotation, scaling, and shearing.

F.3.2 Main training

Main training uses 480p YouTubeVOS and DAVIS data, with temporally ordered frames and consistent sequence-level augmentation. The training setup follows established practices, including greater sampling of higher-quality DAVIS annotations.

  • F.3.2 Main training: 480p YouTubeVOS and DAVIS images are used for main training, with YouTubeVOS resized so the shorter edge has 480 pixels.DAVIS annotations have higher quality and appear five times more often than YouTubeVOS videos.
  • F.3.2 Main training: Each training sample contains three temporally ordered frames from the same video.
  • F.3.2 Main training: Sequence images receive shared initial geometric and appearance transformations, followed by additional per-image color jitter.The shared random seed ensures every image undergoes the same initial transformation.

F.3.3 BL30K training

BL30K is inserted between static-image pretraining and main training to expose the model to complex occlusion patterns. Its training uses modified sampling, extended schedules, and curriculum settings because the synthetic data are harder and introduce domain shift.

  • F.3.3 BL30K training: BL30K is a synthetic VOS dataset used after static-image pretraining and before main training.It is intended to provide complex occlusion patterns unavailable in static images and insufficiently represented in the main datasets.
  • F.3.3 BL30K training: Tiny objects below 100 pixels in the first frame are discarded because they are difficult to learn and unavoidable in the synthetic data.
  • F.3.3 BL30K training: BL30K training runs for 500K iterations after static-image pretraining, followed by extended 300K-iteration main training.Regular training takes around 30 hours, whereas extended training with BL30K takes around four days in total.
  • F.3.3 BL30K training: The learning rate starts at 10^-5 and decays by a ratio of γ = 0.1 at stage-specific iteration points.The decay occurs after 450K iterations for BL30K training and after 250K iterations for its extended main-training stage.
  • F.3.3 BL30K training: BL30K uses a longer temporal-distance annealing schedule because it is relatively harder than the main-training setting.The maximum temporal distance progresses through [5] [10] [20] [25] across the prescribed training percentages.
Loading 2106.05210v2…