PaperlayerView exact revision on arXiv

Source-linked AI summary

Language as Queries for Referring Video Object Segmentation

Jiannan Wu, Yi Jiang, Peize Sun, Zehuan Yuan, Ping Luo

arXiv:2201.00487v2cs.CV

TL;DR

R-VOS requires segmenting a language-referred object across all video frames despite difficult cross-modal reasoning and temporal association. ReferFormer uses language-conditioned queries and dynamic kernels in an end-to-end framework, achieving state-of-the-art results across four benchmarks, including 55.6 J&F on Ref-Youtube-VOS with ResNet-50.

  • Problem

    R-VOS must infer the language-referred object and accurately segment it across frames without first-frame ground-truth masks, requiring comprehensive vision-language understanding.

  • Method

    ReferFormer uses a small set of language-conditioned object queries that become dynamic kernels for mask generation and are linked across frames for tracking.

  • Results

    ReferFormer achieves state-of-the-art performance on Ref-Youtube-VOS, Ref-DAVIS17, A2D-Sentences, and JHMDB-Sentences, including 55.6 J&F with ResNet-50 on Ref-Youtube-VOS.

  • Takeaways & Limitations

    Viewing language as queries provides a unified perspective in which referred-object segmentation and tracking are produced directly without post-processing.

  • Takeaways & Limitations

    Existing R-VOS approaches remain constrained by bottom-up methods' weak instance and temporal reasoning and top-down methods' complex, separately optimized pipelines.

Abstract

from arXiv · show

Referring video object segmentation (R-VOS) is an emerging cross-modal task that aims to segment the target object referred by a language expression in all video frames. In this work, we propose a simple and unified framework built upon Transformer, termed ReferFormer. It views the language as queries and directly attends to the most relevant regions in the video frames. Concretely, we introduce a small set of object queries conditioned on the language as the input to the Transformer. In this manner, all the queries are obligated to find the referred objects only. They are eventually transformed into dynamic kernels which capture the crucial object-level information, and play the role of convolution filters to generate the segmentation masks from feature maps. The object tracking is achieved naturally by linking the corresponding queries across frames. This mechanism greatly simplifies the pipeline and the end-to-end framework is significantly different from the previous methods. Extensive experiments on Ref-Youtube-VOS, Ref-DAVIS17, A2D-Sentences and JHMDB-Sentences show the effectiveness of ReferFormer. On Ref-Youtube-VOS, Refer-Former achieves 55.6J&F with a ResNet-50 backbone without bells and whistles, which exceeds the previous state-of-the-art performance by 8.4 points. In addition, with the strong Swin-Large backbone, ReferFormer achieves the best J&F of 64.2 among all existing methods. Moreover, we show the impressive results of 55.0 mAP and 43.7 mAP on A2D-Sentences andJHMDB-Sentences respectively, which significantly outperforms the previous methods by a large margin. Code is publicly available at https://github.com/wjn922/ReferFormer.

1. Introduction

R-VOS segments language-referred objects across video frames but requires both cross-modal understanding and accurate, temporally consistent segmentation. ReferFormer addresses limitations in bottom-up and top-down pipelines with language-conditioned queries in a unified framework.

  • Task and motivation: R-VOS segments a target object throughout a video from a natural-language description and requires comprehensive vision-language understanding.The task lacks first-frame ground-truth mask annotation, so models must infer the referred object and segment it accurately.
  • Existing pipelines: Bottom-up methods fuse vision and language before decoding masks but lack crucial instance-level information and explicit cross-frame object association.They can encounter discrepancies in predicted objects when scenes change.
  • Existing pipelines: Top-down methods first detect and track candidate objects, but their complex multi-stage pipelines require separately pretrained and fine-tuned networks.Separate optimization across sub-problems can lead to a sub-optimal solution.
  • ReferFormer: ReferFormer conditions a small set of object queries on language so they attend only to the referred object, then converts them into dynamic kernels for mask generation.Its CM-FPN performs multi-level visual-linguistic interaction for fine-grained cross-modal fusion.
  • ReferFormer: ReferFormer links conditional queries across frames to detect, segment, and track the referred object end-to-end without post-processing.The framework also produces classification results and detection boxes.
  • Results: 55.6 J &F is achieved on Ref-Youtube-VOS with a ResNet-50 backbone, exceeding the previous state-of-the-art by 8.4 points.The contribution passage reports this result as evidence of strong benchmark performance.

2. Related Work

Prior R-VOS methods adapt image segmentation or multi-stage detection-and-tracking pipelines, while newer query-based methods pursue end-to-end video reasoning. ReferFormer is presented as a query-based alternative that uses language-conditioned queries to focus on the referred object.

  • Semi-supervised video object segmentation: Semi-supervised video object segmentation propagates first-frame ground-truth masks through a video, commonly using feature matching and memory-based attention.These methods assume an object mask is available in the first frame, unlike R-VOS.
  • Referring video object segmentation: R-VOS replaces the mask reference with language, making the task more challenging than semi-supervised video object segmentation.Bottom-up approaches applied independently to frames can fail to use temporal information and produce inconsistent predictions under scene or appearance variation.
  • Query-based methods: ReferFormer conditions decoder queries on language, duplicates them across frames, and links corresponding query positions to achieve temporal coherence.The queries become instance embeddings and form instance sequences across frames.
  • Referring video object segmentation: Top-down R-VOS methods detect objects in each frame and associate them into video tracklet candidates before grounding language to the candidates.This design addresses temporal association through an explicit multi-stage pipeline.
  • Query-based methods: ReferFormer contrasts with bottom-up and top-down pipelines by using a query-based, unified framework for R-VOS.Its pipeline includes a backbone, Transformer, cross-modal feature pyramid, and instance-sequence segmentation process.
  • Query-based methods: MTTR also uses queries but requires exhaustive segmentation annotations for all objects and supervises un-referred instances during training.Those requirements increase annotation workload and limit practical applicability.

3. Approach

ReferFormer is an end-to-end Transformer framework that conditions a small set of object queries on language to detect, segment, and track the referred object across video frames. It combines language-guided queries, dynamic-kernel mask decoding, and multi-scale cross-modal feature fusion.

  • Inference: At inference, the instance sequence with the highest average reference score is selected, and linked queries provide tracking without post-processing.The selected queries yield the final masks in frame order.
  • Backbone: ReferFormer uses visual and linguistic encoders to construct video feature sequences and text features for cross-modal reasoning.The visual backbone extracts multi-scale features per frame, while RoBERTa produces word-level and sentence-level language features.
  • Language as Queries: Conditional queries use the language expression as decoder guidance, restricting attention to the referred object rather than all objects.The queries are shared across frames and transformed by the decoder into instance embeddings, preserving their relative order across frames.
  • Dynamic Convolution: Dynamic kernels generated from query embeddings act as convolution filters to produce instance masks from corresponding feature maps.The mask head predicts dynamic-kernel parameters, while the box head predicts locations that provide positional priors for kernel application.
  • Cross-modal Feature Pyramid Network: CM-FPN performs visual-language fusion at all pyramid levels to provide multi-scale cross-modal features for segmentation.The pyramid combines encoded-memory features and backbone features across spatial strides from 4× to 32×.
  • Instance Sequence Matching and Loss: Instance sequence matching supervises predictions across frames while maintaining corresponding query positions as object trajectories.Using N queries over T frames produces Nq = T × N predictions, which are matched as instance sequences.

4. Experiments

ReferFormer is evaluated across four referring video object segmentation benchmarks, with comparisons, implementation settings, and ablations covering its performance and design choices. It achieves strong results across datasets and benefits from stronger visual backbones, conditional queries, and multimodal components.

  • Experimental Setup: ReferFormer is evaluated on Ref-Youtube-VOS, Ref-DAVIS17, A2D-Sentences, and JHMDB-Sentences using dataset-specific standard metrics.The experiments use J&F for the first two benchmarks and Precision@K, Overall IoU, Mean IoU, and mAP for the latter two.
  • Main Results: 55.6 J&F with ResNet-50 on Ref-Youtube-VOS exceeds URVOS by 8.4 points and surpasses PMINet’s ensemble result of 54.2.With Swin-Large, the score reaches 62.4 J&F without additional techniques and 64.2 J&F with joint training; Ref-DAVIS17 reaches 58.5 J&F with ResNet-50.
  • Main Results: 55.0 mAP on A2D-Sentences with Video-Swin-Base improves 8.9 mAP over the previous best result.The model also achieves 57.9 for P@0.8 and 21.2 for P@0.9, while outperforming MTTR by 2.5 mAP with a smaller window size.
  • Main Results: ReferFormer significantly outperforms existing methods on JHMDB-Sentences without finetuning, although all methods score low on P@0.9.The authors suggest inaccurate masks generated from human puppets as a possible reason for the low stringent-metric scores.
  • Ablation Study: Performance consistently increases with stronger visual backbones, while post-processing contributes less as backbone strength increases.The reported post-processing gain narrows from +3.8 for ResNet-50 to +0.9 for Swin-Large on J&F.
  • Ablation Study: The model remains effective with one conditional query, saturates at N = 5, and benefits from combining classification, box, and mask costs.Removing either box or mask cost reduces performance; the authors associate mask cost with direct segmentation guidance and box cost with location priors for dynamic kernels.

5. Conclusion

ReferFormer presents a simple, unified R-VOS framework that treats language as queries and directly produces referred-object masks and boxes across video frames. It achieves state-of-the-art performance on four benchmarks.

  • ReferFormer views language as queries that attend only to the referred object, while linking corresponding queries to achieve tracking.
  • The framework directly produces segmentation masks and detected boxes for the referred object in all frames without post-processing.
  • ReferFormer achieves state-of-the-art performance on Ref-Youtube-VOS, Ref-DAVIS17, A2D-Sentences, and JHMDB-Sentences.

A. Additional Dataset Details

The paper evaluates ReferFormer on four R-VOS benchmarks with varied scale, annotation, and split characteristics. Ref-Youtube-VOS is the largest listed dataset, while Ref-DAVIS17 provides descriptions for specific objects.

  • Ref-Youtube-VOS covers 3,978 videos with approximately 15K language descriptions.
  • Ref-Youtube-VOS contains 3,471 training videos with 12,913 expressions and 507 validation videos with 2,096 expressions.
  • Ref-DAVIS17 contains 90 videos, 1,544 expression sentences, and 205 described objects, split into 60 training and 30 validation videos.

B. Additional Implementation Details

The implementation uses AdamW optimization, image-based referring-segmentation pretraining, and task-specific video fine-tuning. Additional joint training mixes Ref-Youtube-VOS with RefCOCO-family data using pseudo video clips.

  • ReferFormer is optimized with AdamW using weight decay 5 × 10^-4, learning rates 5 × 10^-5 for the visual backbone and 10^-4 for other components.
  • The model is pretrained for 12 epochs on Ref-COCO, Ref-COCOg, and RefCOCO+ with T = 1 before video fine-tuning.
  • For A2D-Sentences, the model uses a window size of 5 and six fine-tuning epochs, while JHMDB-Sentences uses the A2D-trained model without fine-tuning.
  • Joint training augments RefCOCO-family images by ±20° to form five-frame pseudo video clips, uses 32 V100 GPUs, and keeps the text encoder frozen.

C. Additional Details of Dynamic Convolution

Dynamic convolution converts instance embeddings into convolutional filters that generate segmentation masks from mask features. Multiple dynamic kernels are implemented efficiently with grouped convolution.

  • A linear projection transforms each instance embedding into dynamic convolutional weights, which filter mask features through consecutive layers.
  • The final dynamic convolutional layer has one output channel and no normalization or activation follows it.
  • For multiple dynamic kernels, the implementation uses group convolution in conv2d for efficiency.

D. Additional Experiment Results

The additional experiment compares class-agnostic and class-discriminative training, finding a clear advantage for the default class-agnostic approach.

  • The default training is class-agnostic, deciding whether an object is referred or not.
  • The class head can be modified by changing the class number to predict the referred object category, enabling class-discriminative training.
  • +2.1 J &F is the performance gain of class-agnostic training over strong class-discriminative training.
  • Table D1 reports the ablation study on class-agnostic training.
Loading 2201.00487v2…