PaperlayerView exact revision on arXiv

Source-linked AI summary

Video Object Segmentation with Language Referring Expressions

Anna Khoreva, Anna Rohrbach, Bernt Schiele

arXiv:1803.08006v3cs.CV

TL;DR

The paper addresses the cost of first-frame pixel masks in video object segmentation by using language referring expressions to identify targets. It temporally stabilizes image-based grounding and uses the resulting boxes to guide mask prediction, achieving mask-level parity on DAVIS16 and scribble-level comparability on DAVIS17.

  • Problem

    First-frame pixel-accurate masks are expensive and time-consuming, motivating a more practical way to identify video segmentation targets.

  • Method

    The method applies image-trained language grounding per frame, enforces temporal consistency on box predictions, and uses the boxes to guide pixel-wise segmentation.

  • Results

    Language supervision performs on par with pixel-mask methods on DAVIS16 and comparably to scribble-based methods on DAVIS17.

  • Takeaways & Limitations

    Language alone can provide high-quality video object segmentation and can complement mask supervision as an additional source of guidance.

  • Takeaways & Limitations

    Videos with multiple similar-looking objects remain challenging because grounding models struggle to preserve object identities across frames.

Abstract

from arXiv · show

Most state-of-the-art semi-supervised video object segmentation methods rely on a pixel-accurate mask of a target object provided for the first frame of a video. However, obtaining a detailed segmentation mask is expensive and time-consuming. In this work we explore an alternative way of identifying a target object, namely by employing language referring expressions. Besides being a more practical and natural way of pointing out a target object, using language specifications can help to avoid drift as well as make the system more robust to complex dynamics and appearance variations. Leveraging recent advances of language grounding models designed for images, we propose an approach to extend them to video data, ensuring temporally coherent predictions. To evaluate our method we augment the popular video object segmentation benchmarks, DAVIS'16 and DAVIS'17 with language descriptions of target objects. We show that our language-supervised approach performs on par with the methods which have access to a pixel-level mask of the target object on DAVIS'16 and is competitive to methods using scribbles on the challenging DAVIS'17 dataset.

1 Introduction

The paper introduces language referring expressions as a practical alternative to first-frame pixel masks for video object segmentation, and evaluates a temporally consistent grounding-and-segmentation approach on augmented DAVIS benchmarks.

  • Motivation: Pixel-accurate first-frame masks enable semi-supervised video object segmentation but are tedious, time-consuming, and costly to annotate.Polygon annotation reportedly takes 79s per instance, while precise delineation can take even longer.
  • Proposed task: Language referring expressions offer a more natural way to specify target objects than pixel masks, point clicks, or scribbles.The paper gives “the man in a red sweatshirt performing breakdance” as an example query.
  • Approach: The approach analyzes image-trained grounding models, enforces temporal coherency, and uses grounded boxes to guide pixel-wise segmentation.It also investigates language as complementary supervision alongside first-frame masks.
  • Evaluation: The authors augment DAVIS16 and DAVIS17 with target-object language descriptions collected from first-frame and full-video viewing settings.Each video receives an average of 7.5 referring expressions, and an expression takes around 5s to provide.
  • Results: On DAVIS16, language supervision performs on par with pixel-mask methods, while on DAVIS17 it is comparable to scribble-based techniques.The paper presents this as evidence that language can provide competitive supervision for video object segmentation.
  • Contributions: The work contributes a language-based segmentation task, augmented benchmarks, grounding analysis, temporal-coherency improvements, and a competitive baseline.It also reports that language descriptions complement visual supervision such as masks.

2 Related Work

Related work spans language grounding in images and video, alongside unsupervised and semi-supervised video object segmentation; this paper reduces first-frame annotation effort with language expressions.

  • Image-domain grounding: Natural-language grounding localizes an expression in an image using a bounding box or segmentation mask.Existing approaches commonly score external object proposals to select the highest-scoring region.
  • Image-domain grounding: The paper experiments with DBNet and MattNet, representing alternative image-based approaches to referring-expression grounding.DBNet uses expression-region classification, whereas MattNet modularly decomposes referring expressions.
  • Video-domain grounding: Video-language research includes object tracking with language and gaze, while the cited concurrent segmentation work focuses on actor/action classes.In contrast, this paper considers arbitrary objects and unconstrained referring expressions.
  • Video object segmentation: Unsupervised video object segmentation uses appearance, motion, proposals, saliency, or optical flow without human input during testing.Convnet approaches cast the task as foreground/background classification using appearance and motion cues.
  • Video object segmentation: Semi-supervised methods provide first-frame masks, clicks, or scribbles and propagate target information through subsequent frames.Methods use superpixels, graphical models, proposals, optical flow, trajectories, or convnets.
  • Annotation efficiency: Because pixel-accurate masks can be prohibitive, the paper follows work using cheaper clicks or scribbles by proposing language supervision.The motivation particularly emphasizes small touchscreen devices.

3 Method

The method grounds a language query to boxes in every frame, temporally smooths those predictions, and refines them into detailed masks using appearance and motion cues.

  • Task formulation: Given a video and textual query, the system aims to produce a pixel-level target-object mask for every frame in which the object appears.The task combines language grounding with video segmentation.
  • Grounding: Image-trained grounding models generate target-object box proposals for every video frame, but off-the-shelf predictions are temporally inconsistent and jittery.DBNet and MattNet are used with Mask R-CNN proposals.
  • Temporal consistency: Temporal consistency re-ranks proposals using inter-frame overlap, temporal distance, original objectness, and grounding matching scores.The re-ranking favors coherent spatio-temporal object tubes and reduces identity switches.
  • Pixel-level segmentation: The segmentation network receives grounded boxes as guidance and recovers foreground/background masks using appearance and motion cues.Boxes are used as rough location and extent information for pixel-wise refinement.
  • Pixel-level segmentation: Boxes, RGB channels, and optical-flow magnitude form a 5-channel input, and the network learns to refine boxes into accurate masks.The model uses ResNet-101 with atrous convolutions and spatial pyramid pooling.
  • Training and inference: Inference uses one per-frame pass without test-time appearance fine-tuning or online adaptation, making the system more efficient.Training uses static saliency images with jittered boxes and synthetic optical flow, followed by DAVIS fine-tuning.
  • Supervision variants: The framework also supports variants combining language with first-frame mask supervision to improve robustness against occlusions and dynamic backgrounds.This establishes flexibility across different supervision sources.

4 Collecting referring expressions for video

The paper augments DAVIS16 and DAVIS17 with referring expressions collected from either the first frame or the full video. Full-video descriptions more often capture actions and are generally more complex, while the resulting resource covers hundreds of objects and videos.

  • The task provides a language expression and requires localizing and segmenting the target object across all video frames.
  • Annotators describe targets either from the first frame alone or after viewing the full video.
  • Full-video expressions mention verbs more often than first-frame expressions, at 44% versus 25%.Their average lengths are 6.3 and 5.5 words, respectively.
  • Adjectives occur in over 50% of annotations, with most describing colors, while full-video expressions also contain more adverbs and prepositions.Spatial or ordering words occur in 6% of first-frame versus 13% of full-video expressions.
  • The augmented DAVIS16/17 resource contains approximately 1.2k expressions for more than 400 objects across 150 videos and approximately 10k frames.

5 Evaluation of natural language grounding in video

The evaluation tests image-trained language grounding models on video and measures localization with bounding-box IoU. Temporal consistency improves unstable framewise predictions, while performance varies with proposals, model, object domain, expression attributes, and scene complexity.

  • The evaluation uses DBNet and MattNet, measuring the IoU of the top-scored box proposal against the ground-truth box across queries.
  • Replacing EdgeBox with Mask R-CNN proposals improves DBNet from 54.1 to 64.9, so subsequent experiments use Mask R-CNN proposals.
  • Temporal consistency improves grounding, including DBNet from 64.9 to 68.8 on DAVIS16 and MattNet from 51.6 to 52.8 on DAVIS17.The image-trained models otherwise produce temporally unstable predictions across frames.
  • MattNet outperforms DBNet on both datasets, with a DAVIS16 comparison of 72.5 versus 68.8.
  • On DAVIS17, grounding drops for non-COCO objects, longer expressions, videos with more objects, and expressions containing verbs for DBNet.MattNet handles verb-containing expressions better than DBNet, at 55.8 versus 37.4.

6 Video object segmentation results

The proposed language-supervised approach achieves competitive video object segmentation on DAVIS16 and DAVIS17, while combining language with masks improves robustness and performance. Ablations and attribute analyses identify temporal consistency, motion cues, and architecture as important components, but grounding remains limiting on challenging multi-object videos.

  • DAVIS16 results: 82.8 mIoU with language alone is on par with pixel-mask supervision at 81.7 on DAVIS16.The method also outperforms point-click alternatives reporting 80.6 and 80.9.
  • DAVIS16 results: 84.5 mIoU from combining mask and language exceeds language-only 82.8 and mask-only 83.1 on DAVIS16.The results support language as complementary guidance alongside visual supervision.
  • Attribute analysis: Language alone is more robust on DAVIS16 videos with low resolution, camera shake, and background clutter, while combined supervision improves robustness across most attributes.The combined model compares favorably with mask-only supervision except for appearance change.
  • Ablation study: Temporal consistency gives a 4.3-point grounding boost, while motion cues and the proposed architecture improve segmentation for moving objects and detailed masks.The ablation reports 80.6 vs. 75.9 with motion cues and a 3.7-point architecture gain, 75.9 vs. 72.2.
  • DAVIS17 results: On DAVIS17, language descriptions significantly underperform mask supervision, but oracle grounding raises performance from 37.3 to 54.9.Ground-truth boxes raise performance from 37.3 to 42.1, implicating grounding and proposal quality in the limitation.
  • DAVIS17 results: The language approach performs on par with scribble supervision on the challenging DAVIS17 benchmark.DAVIS17 remains difficult even for scribble-based supervision.
  • DAVIS17 results: 58 to 59 performance improvement results when language is added to mask supervision on DAVIS17.This again indicates complementarity between the two supervision sources.
  • Qualitative analysis: Qualitative results show successful handling of similar-looking objects, fast motion, deformations, and partial occlusions.The broader conclusion identifies identity preservation among similar objects as a remaining challenge for grounding models.

7 Conclusion

The paper introduces language referring expressions for video object segmentation and extends DAVIS16 and DAVIS17 with textual target descriptions. Experiments indicate that language alone can produce high-quality segmentations, while language guidance can improve robustness and complement visual supervision.

  • The work proposes video object segmentation using language referring expressions and augments DAVIS16 and DAVIS17 with textual descriptions.
  • Language alone can be successfully exploited to obtain high-quality video object segmentations.
  • Language guidance can make segmentation more robust to occlusions, complex dynamics, and cluttered backgrounds.
  • Language descriptions can further improve classical mask-supervised video object segmentation and may be fused with clicks, scribbles, or masks.

Supplementary Material

The supplementary material adds analyses of referring-expression types, grounding models, evaluation metrics, temporal consistency, annotation types, and qualitative comparisons across supervision modes.

  • Section A: Section A compares first-frame and full-video referring expressions and examines when first-frame descriptions become invalid over time.
  • Section B: Section B evaluates natural-language grounding models on Lingual ImageNet Videos and compares them with prior work.
  • Section C: Section C reports additional DAVIS16 metrics and studies grounding models, temporal consistency, annotation types, and qualitative supervision variants.

A Referring expressions for video object segmentation

The benchmarks are augmented with referring expressions collected from either the first frame or the full video. Full-video descriptions are more complex, affect grounding models differently, and only a minority of first-frame descriptions become invalid over time.

  • Dataset construction: All mask-labeled objects in DAVIS16 and DAVIS17 are augmented with non-ambiguous referring expressions.
  • Annotation settings: Annotations describe targets using either the first frame alone or the full video.
  • Annotation settings: Full-video descriptions tend to be longer and more complex because they reflect video-level complexity.
  • Annotation settings: DBNet benefits from full-video descriptions, whereas MattNet struggles with their increased language complexity.
  • Annotation validity: Only ∼15% of first-frame descriptions are marked invalid or ambiguous over time, with mIoU changing from 36.9 to 37.1 after correction.
  • Examples: Collected examples distinguish targets using appearance, spatial position, relationships, actions, and video activities.

B Language grounding results on Lingual ImageNet Videos

The study evaluates language-based object grounding on Lingual ImageNet Videos and compares the proposed temporally consistent approach with prior tracking results.

  • Lingual ImageNet Videos contains referring-expression annotations for 100 videos, split evenly into training and validation sets.Performance is measured using area under the curve (AUC).
  • The comparison includes language-only tracking, tracking initialized with a ground-truth first-frame bounding box, and a combined approach.
  • The proposed method applies temporal consistency to DBNet and MattNet predictions using language input only.
  • Both grounding models significantly outperform the prior method, even when that method receives the ground-truth bounding box on the first frame.

C.1 Additional metrics for DAVIS16

Additional DAVIS16 metrics assess region similarity, contour accuracy, and temporal instability, while DAVIS17 experiments examine grounding models, temporal consistency, and description scope.

  • Additional metrics for DAVIS16: DAVIS16 evaluation uses region similarity J, contour accuracy F, and temporal instability T, with higher J and F and lower T preferred.
  • Additional metrics for DAVIS16: Language-only supervision produces smaller decay for J and F and better overall temporal stability T.
  • Ablations on DAVIS17: MattNet grounding boxes outperform DBNet, with scores of 37.3 versus 35.4.
  • Ablations on DAVIS17: Temporal consistency improves segmentation across grounding models, raising MattNet from 35.4 to 37.3 and DBNet from 32.6 to 35.4.
  • Ablations on DAVIS17: Full-video descriptions marginally improve DBNet from 35.4 to 35.5 but reduce MattNet from 37.3 to 35.5.

C.3 Qualitative results for video object segmentation

Qualitative results show language-only segmentation handling substantial motion and occlusion, while comparisons examine robustness against mask supervision and multi-instance segmentation with combined supervision.

  • Language-only supervision: Language-only supervision successfully handles shape deformations, fast motion, and partial or full occlusions.
  • Mask plus language supervision: Mask plus language supervision produces high-quality instance-level segmentation of multiple similar-looking objects on DAVIS17.
  • Language versus mask supervision: Language-only supervision is reported as more robust than pixel-level mask supervision for similar-looking instances and camera-view changes.
  • Referring expressions: The qualitative examples include expressions identifying people, vehicles, animals, and fish by appearance, position, size, or activity.
  • Referring expressions: Additional examples distinguish goldfish using relative size, color, position, and proximity to other fish.
Loading 1803.08006v3…