PaperlayerView exact revision on arXiv

Source-linked AI summary

Tracking Anything with Decoupled Video Segmentation

Ho Kei Cheng, Seoung Wug Oh, Brian Price, Alexander Schwing, Joon-Young Lee

arXiv:2309.03903v1cs.CV

TL;DR

Expensive video annotations limit extending end-to-end segmentation to new, especially large-vocabulary, tasks. DEVA decouples task-specific image segmentation from reusable class-agnostic bi-directional propagation, and reports favorable or state-of-the-art performance across several data-scarce settings. Its scope is narrower when new objects must be detected by the temporal model alone or when video training data is sufficient.

  • Problem

    End-to-end video segmentation requires costly annotated video data and its scalability to large-vocabulary or open-world settings is questionable.

  • Method

    DEVA combines a target-task image segmentation model with a universal class-agnostic bi-directional temporal propagation model that denoises and fuses segmentation hypotheses.

  • Results

    DEVA achieves state-of-the-art results on large-scale video panoptic and open-world video segmentation and competitive results on referring and unsupervised video object segmentation.

  • Takeaways & Limitations

    Decoupling enables reuse of external task-agnostic propagation data and reduces reliance on target-task video annotations in large-scale or open-world settings.

  • Takeaways & Limitations

    The task-agnostic temporal model cannot detect new objects by itself, and end-to-end approaches work better when sufficient video training data is available.

Abstract

from arXiv · show

Training data for video segmentation are expensive to annotate. This impedes extensions of end-to-end algorithms to new video segmentation tasks, especially in large-vocabulary settings. To 'track anything' without training on video data for every individual task, we develop a decoupled video segmentation approach (DEVA), composed of task-specific image-level segmentation and class/task-agnostic bi-directional temporal propagation. Due to this design, we only need an image-level model for the target task (which is cheaper to train) and a universal temporal propagation model which is trained once and generalizes across tasks. To effectively combine these two modules, we use bi-directional propagation for (semi-)online fusion of segmentation hypotheses from different frames to generate a coherent segmentation. We show that this decoupled formulation compares favorably to end-to-end approaches in several data-scarce tasks including large-vocabulary video panoptic segmentation, open-world video segmentation, referring video segmentation, and unsupervised video object segmentation. Code is available at: https://hkchengrex.github.io/Tracking-Anything-with-DEVA

1. Introduction

DEVA addresses the difficulty of scaling end-to-end video segmentation to large-vocabulary and data-scarce settings by separating task-specific image segmentation from task-agnostic temporal propagation. Its bi-directional propagation denoises and fuses segmentation hypotheses, supporting favorable results across several video segmentation tasks.

  • Results: >60% relative VPQ improvement occurs for rare classes with small amounts of target-domain training data.Figure 2 attributes this gain to using external class-agnostic temporal propagation data unavailable to existing end-to-end baselines.
  • Motivation: End-to-end video segmentation has advanced on small-vocabulary benchmarks, but its scalability to large-vocabulary and open-world data remains questionable.YouTube-VIS contains 40 categories and Cityscape-VPS 19, whereas VIPSeg contains 124 classes and is substantially more difficult under the same backbone.
  • Approach: DEVA separates task-specific image-level segmentation from task-agnostic temporal propagation to reduce reliance on target-domain video annotations.The image-level model is cheaper to train for the target task, while the universal temporal model is trained once and generalized across tasks.
  • Approach: Bi-directional propagation denoises image segmentations through in-clip consensus and combines image-level and temporally propagated results.The method performs (semi-)online fusion of hypotheses from different frames to produce temporally coherent segmentations.
  • Results: DEVA achieves state-of-the-art results on large-scale video panoptic and open-world video segmentation, and competitive results on referring and unsupervised video object segmentation.The paper frames the method as a strong baseline when image models are available but video data is scarce, rather than a replacement for specialized end-to-end approaches.

2. Related Works

Prior work largely relies on end-to-end training or tracking-by-detection, which is costly or sensitive to image-level errors in broad-vocabulary video settings. DEVA instead unifies task-specific image segmentation with universal, bi-directional propagation to denoise and extend segmentation across videos.

  • End-to-End Video Segmentation: End-to-end video methods require expensive target-task training, making their scalability to larger vocabularies with limited annotations questionable.MaskProp still requires end-to-end target-task training because its propagation is not class-agnostic.
  • Open-World Video Segmentation: Open-world video segmentation evaluates common and uncommon classes separately across 482 object classes in diverse scenarios.BURST baselines use COCO-trained image proposals followed by frame-by-frame association with box IoU or STCN, while OWTB adds optical flow and Re-ID features.
  • Decoupled Video Segmentation: Tracking-by-detection treats image-level detections as immutable and depends heavily on their quality, making it sensitive to image-level errors.DEVA differs by using bi-directional propagation to denoise image segmentations and support longer-term associations.
  • Video Object Segmentation: Semi-supervised VOS cannot account for errors in the initial segmentation or incorporate new image-model segmentations at later frames.The paper positions bi-directional propagation as the mechanism that unifies image segmentation and propagation models.
  • Segmenting/Tracking Anything: Universal image segmentation extensions to video rely on single-frame segmentation and lack DEVA's in-clip-consensus denoising capability.Other unified video frameworks train across target tasks end-to-end, while DEVA trains only the target image model and reuses universal temporal propagation.

3. Decoupled Video Segmentation

The framework separates task-specific image segmentation from universal temporal propagation, then uses in-clip consensus and fusion to produce coherent video segmentations.

  • Decoupled Video Segmentation: Decoupled video segmentation uses a swappable image model for task-specific hypotheses and a class-agnostic propagation model to associate them across the video.The propagation model operates on segmented-frame memory and a query image, while the image model can be replaced for different target tasks.
  • Overview: The pipeline initializes from the first frame, denoises image-level predictions with near-future in-clip consensus, and propagates the result temporally.In-clip consensus uses a small future clip in the semi-online setting; the online setting uses n = 1 and Ct = Segt.
  • In-clip Consensus: In-clip consensus aligns proposals to the target frame, pools them, and selects a non-overlapping subset favoring mutually supported segments while penalizing unsupported or noisy ones.Pairwise support is based on IoU, with proposals supporting one another excluded from simultaneous selection; the penalty weight is α = 0.5.
  • In-clip Consensus: The consensus output is formed from proposals whose optimized indicator variables equal one, with overlapping selected segments resolved by prioritizing smaller segments.The optimization maximizes support and penalty terms subject to a zero-overlap constraint among selected proposals.
  • Merging Propagation and Consensus: The method merges propagated and consensus segmentations by unioning associated pairs while passing unassociated segments through to the final output.Overlapping output segments are resolved by prioritizing smaller segments.

4. Experiments

Experiments evaluate the decoupled approach across large-vocabulary, open-world, referring, and unsupervised video segmentation, including sensitivity to target-domain training data. Results favor decoupled propagation for long-term associations and data-scarce settings, while specialized end-to-end methods remain stronger when video data is sufficient.

  • Discussion and Limitations: The method is positioned as a strong baseline when image models are available but video data is scarce, not as a replacement for specialized end-to-end approaches.The paper states that specialized end-to-end frameworks outperform it when sufficient video-level training data is available.
  • Large-Scale Video Panoptic Segmentation: Figure 5 shows that the decoupled approach decreases more slowly than Video-K-Net as k increases, indicating better long-term propagation.
  • Open-World Video Segmentation: The method performs better on BURST for both common and uncommon classes, using Mask2Former for common classes and EntitySeg for uncommon classes.EntitySeg works better for novel objects, and image segmentation backbones can be switched without finetuning.
  • Large-Scale Video Panoptic Segmentation: On VIPSeg, the method outperforms Clip-PanoFCN and Video-K-Net, especially at large k where long-term associations matter.Performance gains at large k support the advantage of decoupling temporal propagation from end-to-end training.
  • Open-World Video Segmentation: In BURST comparisons, the approach outperforms tracking-by-detection baselines by improving image segmentations and enabling long-term tracking.The baselines associate per-frame detections with short-term temporal modules, making them sensitive to image-level misdetections.
  • Referring and Unsupervised Video Object Segmentation: The approach also reports favorable results on referring video segmentation and unsupervised video object segmentation datasets.Referring segmentation uses J &F on Ref-DAVIS17 and Ref-YouTubeVOS; unsupervised evaluation covers DAVIS-16 and DAVIS-17.
  • Varying Training Data: With little VIPSeg training data, the decoupled model achieves a higher relative VPQ improvement over Video-K-Net for rare classes.The temporal propagation model remains unchanged because it uses no target-domain data.

5. Conclusion

DEVA is a decoupled video segmentation approach that combines image segmentation with bi-directional temporal propagation to extend image segmentation to video. Its propagation system uses memory-based components adapted from XMem and is designed for task-agnostic generalization.

  • DEVA combines task-specific image segmentation with bi-directional propagation to track objects across video.The approach leverages external task-agnostic data to reduce reliance on target-task annotations.
  • The propagation model takes segmented frames in memory and a query image, then segments the query frame using objects stored in memory.The memory is a compact representation computed from previously segmented frames.
  • DEVA adopts XMem’s online feature-memory design, which reads incoming query representations from an internal memory.The implementation makes technical modifications to XMem for robustness in the generalized setting.
  • The temporal propagation network contains a key encoder, value encoder, mask decoder, and Conv-GRU.The key and value encoders construct representations for memory reading and object-mask prediction.
  • The key encoder produces multi-scale image features and a key used for memory querying, while the value encoder combines images with object masks.The key encoder uses ResNet-50 features; the value encoder uses ResNet-18 features.
  • The mask decoder converts memory readouts and skip connections into object masks, while the Conv-GRU updates the hidden state.The decoder uses upsampling blocks with residual fusion of skip-connected features.
  • The feature memory comprises sensory, working, and long-term memories that preserve temporal information at different update frequencies.Sensory memory updates every frame; working memory stores key-value pairs and consolidates older entries into long-term memory.
  • The working-memory update interval is synchronized with the in-clip consensus frequency, with Tmax = 10, Tmin = 5, and top-k filtering k = 30.The long-term memory can contain up to 10,000 elements.

A.5. Training

The temporal propagation model is pretrained on synthesized clips from static image segmentation data and then trained on video datasets. Training modifications target robustness, temporal consistency, and stability.

  • XMem is pretrained on static image segmentation datasets by synthesizing three-frame clips with affine and thin-spline deformations.
  • The model is then trained on YouTubeVOS and DAVIS using clips of length eight.
  • DEVA introduces three training modifications to improve robustness.The modifications are evaluated as changes to the temporal propagation training process.
  • OVIS is added because training on DAVIS and YouTubeVOS had produced nearly perfect segmentation results.
  • Stable augmentation applies the same rotation and crop throughout a video sequence to improve temporal consistency.Applying different transformations within a sequence encourages appearance invariance but harms temporal-information learning.
  • Gradient norms are clipped at 3.0, which the authors find leads to faster convergence and more stable training.
  • Training uses batch size 16, AdamW, and learning rates of 2e-5 during pretraining and 1e-5 during main training.The learning rate decays by 0.1 at the 120,000-th and 140,000-th iterations.

A.6. Video Object Segmentation Evaluation

The temporal propagation model is evaluated on three video object segmentation benchmarks using standardized input resizing and available-frame processing. Results are summarized in Table S1.

  • The evaluation covers DAVIS-2017 validation/test-dev, YouTubeVOS-2019 validation, and MOSE validation.
  • Inputs are resized so the shorter side is 480px, and outputs are bilinearly upsampled to the original resolution.
  • All available YouTubeVOS frames are used by default, and Table S1 reports the benchmark comparisons.

A.7.1 On VOS Tasks

The evaluation examines temporal propagation on standard and converted video object segmentation datasets, then tests transfer to video panoptic segmentation. It emphasizes whether training and design modifications improve generalization across settings.

  • On VOS Tasks: The VOS evaluation uses DAVIS and converts OVIS and UVO validation sets into the VOS format.Segments absent from the first frame are discarded, and first-frame ground-truth masks are provided as input.
  • On VOS Tasks: Stable augmentation keeps transformations consistent within sequences, encouraging temporal information rather than only invariant appearance modeling.Figure S1 contrasts fixed and varying crops or rotations across frames.
  • On VOS Tasks: The study compares XMem with modified variants using J &F performance comparisons on VOS tasks.
  • On Large-Scale Video Panoptic Segmentation: The transfer experiment evaluates temporal-propagation settings on VIPSeg with a Mask2Former-R50 image model in a semi-online protocol.Table S3 reports the performance comparisons.
  • On VOS Tasks: Figure S2 shows that stable augmentation tracks all nearly identical ants, whereas aggressive augmentation loses several ants toward the sequence end.The comparison concerns the yellow, blue, and cyan ants.
  • On Large-Scale Video Panoptic Segmentation: The overall framework matters more than particular temporal-propagation design choices, since the method still works well without the proposed propagation modifications.

B.1. Sensitivity to Training Data

The study tests how class-agnostic temporal-propagation training data affect decoupled video segmentation, including the amount of data and category overlap with the target task.

  • Training-data scale: The temporal propagation model is trained on class-agnostic image-segmentation and mask-propagation data that are cheap to access because they require no class-specific annotations.The experiment varies the amount of class-agnostic video-level training data under two settings.
  • Category overlap: The study separately investigates whether overlap between propagation-training categories and target-task categories affects final performance.This tests overlap between temporal-propagation training data and the target task rather than changing the target image model.
  • Category-overlap protocol: Using only YouTubeVOS for temporal training provides 65 annotated categories, which are partitioned into overlapping, non-overlapping, and ambiguous VIPSeg classes for separate evaluation.The analysis ignores the ambiguous set when comparing final performance across the other two groups.

C. Detailed Experimental Settings and Results

The experiments specify datasets, task-specific image models, propagation protocols, consensus construction, labeling, and runtime across several video-segmentation settings.

  • Video panoptic segmentation: VIPSeg evaluation uses its 720p version, validation set of 343 videos, and official VPQ/STQ computation.Videos are downsampled during propagation and restored to their original resolution afterward.
  • Video panoptic segmentation: For video panoptic segmentation, class labels come from the image model and online majority voting selects each propagated object's output label.Labels are accumulated when propagated segments match in-clip consensus segments.
  • Runtime: 4.0fps is achieved by the decoupled method versus 4.9fps for Video-K-Net, while its VPQ is 52% higher and runtime is 18% slower.The default semi-online setting uses three-frame clips, merges every five frames, and invokes the image model on 60% of frames.
  • Open-world video segmentation: BURST evaluation covers 993 validation and 1,421 test videos, processing every third frame because ground truth is annotated sparsely.Propagation uses 480px shortest-side videos with bilinear upsampling to the original resolution.
  • Image models: Mask2Former and EntitySeg use author-provided Swin-L COCO checkpoints, with Mask2Former post-processing resolving overlapping panoptic predictions.These image models support the open-world evaluation settings.
  • Consensus and referring segmentation: Offline consensus selects the highest-confidence frame among ten uniformly spaced frames, combines soft probability maps by confidence-derived weights, and propagates forward and backward from the keyframe.Referring segmentation uses ReferFormer, while known language expressions eliminate the need for integer-program association across frames.

D. Results on YouTube-VIS

On small-vocabulary YouTube-VIS, specialized end-to-end approaches perform better because sufficient video-level training data are available.

  • YouTube-VIS results: Specialized end-to-end methods outperform the proposed approach on YouTube-VIS because this dataset provides sufficient training data.The paper presents these results as additional experiments and uses an online video panoptic segmentation setting.

E. Qualitative Results

Qualitative and comparison results cover panoptic, open-world, referring, and additional video-segmentation settings using task-specific image models and reported baselines.

  • Visualization: Qualitative visualizations track object identities with unique colors and use overlays combining colored segmentation with the input image.Color changes indicate that an object's identity changed, potentially reflecting instability such as flicker.
  • Video panoptic segmentation: The VIPSeg comparison uses Mask2Former with semi-online propagation and compares against Video-K-Net on validation videos.The cited table reports performance comparisons for the large-scale video panoptic setting.
  • Open-world video segmentation: The open-world comparison uses EntitySeg with semi-online propagation against Mask2Former plus an STCN tracker on BURST and Internet videos.Extended BURST validation and test results are reported in Table S5.
  • Referring video segmentation: Referring video segmentation uses X-Decoder in an offline setting to examine open-world scenarios beyond standard Ref-DAVIS and Ref-YouTubeVOS.The formulation can switch image backbones and incorporate newer image-segmentation models.
Loading 2309.03903v1…