Source-linked AI summary
End-to-End Referring Video Object Segmentation with Multimodal Transformers
Adam Botach, Evgenii Zheltonozhskii, Chaim Baskin
TL;DR
RVOS requires segmenting text-referred objects across video while handling action descriptions and tracking, and existing solutions often rely on complicated pipelines. MTTR addresses this with an end-to-end multimodal Transformer that predicts object sequences before selecting the referred instance. It simplifies the pipeline and outperforms prior methods across benchmarks, including a 5.7 mAP improvement on A2D-Sentences.
Problem
RVOS must segment text-referred objects across frames while handling action-based descriptions and cross-frame tracking, but existing approaches typically use complicated pipelines.
Method
MTTR is an end-to-end multimodal Transformer that models RVOS as sequence prediction, producing predictions for all objects before selecting the text-referred one.
Results
MTTR outperforms existing methods across standard benchmarks, including a 5.7 mAP improvement on A2D-Sentences.
Takeaways & Limitations
A single multimodal Transformer can simplify RVOS while achieving strong benchmark performance across multiple datasets and metrics.
Takeaways & Limitations
The matching formulation assumes more predicted sequences than annotated instances and pads unmatched ground-truth slots with no-object entries.
Abstract
from arXiv · showhide
The referring video object segmentation task (RVOS) involves segmentation of a text-referred object instance in the frames of a given video. Due to the complex nature of this multimodal task, which combines text reasoning, video understanding, instance segmentation and tracking, existing approaches typically rely on sophisticated pipelines in order to tackle it. In this paper, we propose a simple Transformer-based approach to RVOS. Our framework, termed Multimodal Tracking Transformer (MTTR), models the RVOS task as a sequence prediction problem. Following recent advancements in computer vision and natural language processing, MTTR is based on the realization that video and text can be processed together effectively and elegantly by a single multimodal Transformer model. MTTR is end-to-end trainable, free of text-related inductive bias components and requires no additional mask-refinement post-processing steps. As such, it simplifies the RVOS pipeline considerably compared to existing methods. Evaluation on standard benchmarks reveals that MTTR significantly outperforms previous art across multiple metrics. In particular, MTTR shows impressive +5.7 and +5.0 mAP gains on the A2D-Sentences and JHMDB-Sentences datasets respectively, while processing 76 frames per second. In addition, we report strong results on the public validation set of Refer-YouTube-VOS, a more challenging RVOS dataset that has yet to receive the attention of researchers. The code to reproduce our experiments is available at https://github.com/mttr2021/MTTR
1. Introduction
RVOS requires jointly reasoning over text, video, segmentation, and tracking, yet existing methods use complicated pipelines. MTTR instead uses a single multimodal Transformer and sequence prediction to simplify the task while improving benchmark performance.
- Motivation: RVOS segments a text-referred object across video frames, where references may describe appearance or actions.Action-based expressions often cannot be inferred from one frame.
- Motivation: Tracking is needed to associate the referred object across frames despite occlusions or motion blur.
- Approach: MTTR uses a single multimodal Transformer to process text and video together, without text-related inductive-bias modules or additional mask refinement.
- Approach: MTTR models RVOS as parallel sequence prediction, generating predictions for all video objects before selecting the text-referred instance.
- Evaluation: MTTR significantly outperforms existing methods across metrics on A2D-Sentences and JHMDB-Sentences, with strong results on Refer-YouTube-VOS.Refer-YouTube-VOS is described as a challenging dataset that has received limited attention.
2. Related Work
Prior RVOS systems use complicated pipelines to align visual, temporal, and linguistic information. Related Transformer work motivates MTTR’s multimodal, sequence-based design, while existing multimodal detection methods remain limited to static images or unavailable annotation types.
- Referring video object segmentation: State-of-the-art RVOS approaches typically use complicated pipelines to aggregate and align visual, temporal, and linguistic information.
- Transformers: Transformers model dependencies between sequence inputs and outputs through attention, supporting their use in multimodal sequence processing.
- Transformers: DETR uses learned object queries to reason about global image context and output detection predictions in parallel, simplifying traditional detection pipelines.
- Transformers: Video Swin extends hierarchical Transformers to video and can be modified to produce per-frame feature maps for video segmentation.
- Multimodal detection: MDETR detects text-conditioned objects in static images, whereas RVOS lacks the aligned text-and-box annotations on which its performance largely depends.
3. Method
The method frames RVOS as sequence prediction with a single multimodal Transformer that jointly processes video and text, produces tracked instance sequences, and selects the sequence most associated with the referring expression.
- Task definition: RVOS inputs a frame sequence and text query, then segments the referred object in each annotated frame of interest.Annotated frames may cover only a subset of the video because producing mask annotations is costly.
- Feature extraction: Video and linguistic features are extracted separately, projected to a shared dimension, and combined into per-frame multimodal sequences.The visual encoder is spatio-temporal, while the text encoder is Transformer-based.
- Instance prediction: A Transformer decoder uses shared-weight object queries across frames to form instance sequences, enabling natural tracking of each object instance.The queries attend to entity-related information in multimodal sequences and are trained to represent the same instance in designated frames.
- Temporal encoder: Video Swin is used as the temporal encoder because its limited temporal downsampling supports per-frame feature maps for full-sequence segmentation.The paper contrasts this with I3D, which was designed for action classification and is less suited to fine-detail instance segmentation.
- Segmentation: Each instance sequence generates conditional segmentation kernels that produce mask sequences from high-resolution frame features, followed by upsampling to ground-truth resolution.An FPN-like spatial decoder hierarchically fuses Transformer and temporal-encoder features before mask generation.
- Matching, loss, and inference: The model trains with Hungarian matching and combines mask alignment with reference supervision, while inference selects the sequence with the highest temporal segment voting score.Temporal segment voting aggregates associations with the text-referred object, emphasizing relevant visible portions and disregarding irrelevant or occluded portions.
4. Experiments
MTTR is evaluated on three RVOS benchmarks using standard metrics, comparisons with prior methods, and ablations of temporal, visual, and textual components. It achieves strong benchmark performance and remains effective under challenging video conditions.
- Evaluation setup: MTTR is evaluated on A2D-Sentences, JHMDB-Sentences, and Refer-YouTube-VOS using IoU, precision@K, and mAP-based measures.A2D-Sentences and JHMDB-Sentences use Overall IoU, Mean IoU, precision@K, and mAP; Refer-YouTube-VOS uses a public validation setting.
- Evaluation setup: The authors identify inconsistencies in prior mAP calculations and recommend the COCO API for fair evaluation.Their released implementation uses the COCO API to calculate mAP.
- Comparison with state-of-the-art methods: 5.7 mAP gain and 6.7% absolute improvements on Mean and Overall IoU are achieved on A2D-Sentences with w = 10, while processing 76 frames per second.The same configuration is reported to outperform existing methods across metrics; a window of size 8 is used for fair comparison with prior works.
- Comparison with state-of-the-art methods: MTTR outperforms existing approaches without fine-tuning on JHMDB-Sentences, although all methods obtain low P@0.9 because of imprecise mask annotations.The evaluation uniformly samples three frames from each video.
- Comparison with state-of-the-art methods: MTTR outperforms methods evaluated on the full Refer-YouTube-VOS dataset despite using less data and only the more challenging full-video-expression subset.Its performance is also competitive with leading 2021 RVOS competition methods, which use ensembles and additional datasets.
- Ablation studies: MTTR surpasses CMPC-I by 6.1 mAP and 8.7% absolute Mean IoU using only target frames, while Transformer-based text encoders perform comparably and simpler embeddings perform slightly worse.The temporal-encoder comparison uses DeepLab-ResNet101 pretrained on PASCAL-VOC; the text-encoder comparison includes BERT-base, Distill-RoBERTa-base, GloVe, and fastText.
- Ablation studies: Increasing temporal context from 1 to 10 frames yields a 4.3 mAP gain and 3.7% absolute Mean IoU improvement, but w = 12 reduces performance.The authors attribute larger-context gains to better extraction of action-related information.
- Qualitative results: Visual examples show MTTR tracking and segmenting referred objects amid similar instances, occlusions, or periods when the target is outside the frame.The temporal segment voting example illustrates how predictions across a sequence can select the correct zebra when it is initially not visible.
5. Conclusion
MTTR is a simple, end-to-end Transformer approach that processes text and video frames jointly and models RVOS as sequence prediction. On standard benchmarks, it outperforms existing state-of-the-art methods by a large margin.
- MTTR processes text and video frames simultaneously in a single multimodal Transformer.
- MTTR models referring video object segmentation as a sequence prediction problem.
- 5.7 mAP improvement on A2D-Sentences demonstrates MTTR’s reported margin over existing state-of-the-art methods.
- The authors position MTTR as evidence of Transformers’ potential for complex multimodal tasks.
A. Additional Method Details
The additional method details define the Dice and Focal cost and loss functions used for mask-sequence supervision. They specify pixel-level quantities, smoothing, and Focal-loss hyperparameters.
- The section defines Dice and Focal cost and loss functions for the mask sequences used in the method.
- The Dice coefficient is introduced for comparing two segmentation masks, mA and mB.
- A smoothing constant s = 1 is added to the Dice expression’s numerator and denominator to avoid division by 0.
- Dice cost and Dice loss are defined between ground-truth and predicted mask sequences.
- The Focal loss is defined between corresponding predicted and ground-truth segmentation masks at each time step.
- The implementation uses α = 0.25 and γ = 2 for the Focal loss.
B. Additional Dataset Details
The paper evaluates on A2D-Sentences, JHMDB-Sentences, and Refer-YouTube-VOS, whose additional details describe their video, expression, object, actor, and action coverage.
- A2D-Sentences contains 3,754 videos, 7 actor classes, 8 action classes, and 6,655 sentences describing actors and their actions.
- JHMDB-Sentences contains 928 videos and 928 corresponding sentences covering 21 action classes.
- Refer-YouTube-VOS contains 27,899 text expressions for 7,451 objects in 3,975 videos across 94 common categories.
- Refer-YouTube-VOS provides separate first-frame and full-video expression subsets with training and validation splits.
C. Additional Implementation Details
The implementation preserves per-frame video embeddings, uses a compact Transformer with fixed query and positional-encoding choices, and decodes segmentation through an FPN-like module.
- The Video Swin Transformer’s temporal down-sampling layer is removed so the multimodal Transformer receives per-frame embeddings.
- Summing pretrained convolution weights over the temporal dimension produces a 1 × 4 × 4 kernel while retaining support for Kinetics-400 pretrained weights.
- The Transformer decoder receives Nq = 50 object queries per input frame and uses three encoder and three decoder layers for efficiency.
- Fixed sine spatial positional encodings are added to each frame’s features before Transformer processing.
- The spatial decoder GSeg is an FPN-like module built from 2D convolutions, GroupNorm, ReLU layers, and nearest-neighbor upsampling.
- Segmentation kernels and FSeg feature maps use dimension Ds = 8.
C.4. Additional Training Details
The training setup specifies optimization choices, data augmentation, auxiliary losses, and dataset-specific schedules for MTTR.
- MTTR uses AdamW with weight decay 10^-4, gradient clipping at 0.1, separate learning rates, and a frozen text encoder.
- Auxiliary decoding losses on every Transformer decoder layer expedite training and improve overall performance.
- Horizontal frame flips are paired with swaps of direction-related words to enhance position awareness during training.
- A2D-Sentences training lasts 70 epochs, uses window size w = 8 and batch size 6, and takes about 31 hours on 3 RTX 3090 GPUs.
- The ablation is identified as an ablation on the number of object queries.
- Refer-YouTube-VOS training uses window size w = 12 and batch size 4, taking about 45 hours on 4 A6000 GPUs.
D.1. Ablations
The object-query ablation evaluates MTTR on A2D-Sentences with window size w = 6 and finds its best performance at Nq = 50.
- Nq = 50 achieves the best performance in the object-query ablation.Lower Nq values may provide insufficiently diverse object queries, whereas higher values may require a longer
- The experiment trains and evaluates MTTR on A2D-Sentences with window size w = 6 across different numbers of object queries.
D.2. Analysis of the Effect of TSVS
The TSVS analysis shows how grouping predictions across temporal segments helps select the referred zebra consistently across frames.
- Without TSVS, frame-wise selection can choose visible but incorrect zebras before the referred zebra appears.
- With TSVS, predictions of the correct zebra vote together across the final three frames, producing high reference scores for its sequence.
- The selected sequence segments only the correct zebra throughout the video, leaving the first two frames unsegmented when it is not visible.