PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning What to Learn for Video Object Segmentation

Goutam Bhat, Felix Järemo Lawin, Martin Danelljan, Andreas Robinson, Michael Felsberg, Luc Van Gool, Radu Timofte

arXiv:2003.11540v2cs.CV

TL;DR

Video object segmentation must identify targets in later frames from only a first-frame reference mask, so capturing limited target information is fundamental. The paper introduces an end-to-end VOS architecture with a differentiable few-shot learner that learns what target representation to produce. It achieves an overall score of 81.5 on YouTube-VOS 2018, a 2.6% relative improvement over the previous best result.

  • Problem

    VOS must segment target objects in subsequent frames using only a first-frame reference mask, making effective capture of scarce target information a fundamental research question.

  • Method

    The architecture uses a differentiable few-shot learner to predict a parametric target model and learns the labels and importance weights that define its training objective.

  • Results

    81.5 overall score on YouTube-VOS 2018 delivers a 2.6% relative improvement over the previous best result and establishes a new state of the art.

  • Takeaways & Limitations

    Learning what the internal few-shot learner should learn enables a rich target representation that guides the segmentation network toward accurate masks.

  • Takeaways & Limitations

    The internal learner usually has only one ground-truth annotated frame in its few-shot training set, although additional frames may be included through self-annotation.

Abstract

from arXiv · show

Video object segmentation (VOS) is a highly challenging problem, since the target object is only defined during inference with a given first-frame reference mask. The problem of how to capture and utilize this limited target information remains a fundamental research question. We address this by introducing an end-to-end trainable VOS architecture that integrates a differentiable few-shot learning module. This internal learner is designed to predict a powerful parametric model of the target by minimizing a segmentation error in the first frame. We further go beyond standard few-shot learning techniques by learning what the few-shot learner should learn. This allows us to achieve a rich internal representation of the target in the current frame, significantly increasing the segmentation accuracy of our approach. We perform extensive experiments on multiple benchmarks. Our approach sets a new state-of-the-art on the large-scale YouTube-VOS 2018 dataset by achieving an overall score of 81.5, corresponding to a 2.6% relative improvement over the previous best result.

1 Introduction

VOS must segment target objects from later video frames using only a first-frame reference mask, making target-information capture central. The paper introduces an end-to-end architecture with a differentiable few-shot learner that learns richer target representations and achieves a new YouTube-VOS benchmark state of the art.

  • Motivation: VOS identifies target objects pixel-wise in video sequences from only a first-frame reference segmentation.This limited target information makes capturing and using the target a central challenge.
  • Approach: The approach encodes target information in a compact parametric model predicted by a differentiable few-shot learner from the first-frame annotation.The target model produces an intermediate mask representation that the segmentation decoder uses for final predictions.
  • Learning What to Learn: The learner is trained to learn ground-truth labels and spatial importance weights, enabling a richer target representation than a single-channel mask.This addresses the limitation of standard few-shot learning, which directly predicts only the object mask.
  • Training: All modules are trained end-to-end by maximizing segmentation accuracy on annotated VOS videos.Differentiability allows the internal learner and the rest of the architecture to be optimized jointly.
  • Results: 81.5 overall score on YouTube-VOS 2018 establishes a new state of the art, a 2.6% relative improvement over the previous best result.The paper evaluates on YouTube-VOS and DAVIS and includes ablations of the proposed components.

2 Related Work

Related VOS methods differ mainly in how they encode and exploit target-specific information. Prior approaches include online adaptation, target representations, feature matching, memory attention, and correlation filters.

  • Target Models in VOS: Early VOS methods adapted semantic segmentation networks through online fine-tuning, but this could overfit the initial target appearance and require impractically long runtimes.More recent methods therefore integrate target-specific appearance models into the segmentation architecture.
  • Target Representations: RGMP encodes the reference frame with its ground-truth target mask and concatenates the resulting representation with current-frame features.An extension processes multiple targets using an instance-specific attention generator.
  • Matching and Memory: Other methods store foreground features for feature matching, retrieve encoded information through space-time memory attention, or predict template correlation filters from the input target mask.These approaches represent alternative strategies for exploiting target information during segmentation.

3 Method

The method treats video object segmentation as an internal few-shot learning problem: a target model is inferred from the first-frame annotation and then used to condition segmentation in later frames. It learns richer target representations and weighted objectives within an efficient, differentiable, end-to-end architecture.

  • Video Object Segmentation as Few-shot Learning: VOS infers target-model parameters from the first image and mask, then combines the target module's output with current-image features for target-aware segmentation.The target model captures target information while the segmentation network produces the final mask.
  • Internal Learner: The differentiable few-shot learner minimizes first-frame segmentation error to produce a powerful parametric target model while remaining efficient enough for end-to-end training.The optimization-based formulation is designed to be both differentiable and practical for video processing.
  • Learning What to Learn: Learning only a single-channel mask limits target-specific information and makes full segmentation difficult, motivating a richer target representation.The segmentation network can benefit from information beyond an approximate target mask.
  • Learning What to Learn: A trainable label generator encodes the first-frame mask into potentially multi-dimensional targets for the few-shot learner, enabling richer cues for the segmentation network.The encoded labels replace direct use of the first-frame mask as the learner's supervision target.
  • Learning What to Learn: A weight predictor assigns element-wise importance to the generated labels, helping the learner focus on crucial target properties despite data imbalance.The example motivation is that sparse boundary pixels are difficult to learn without reweighting.
  • Internal Learner: The target module is a linear convolutional map from deep image features to a spatially aligned target-aware encoding, chosen for efficiency and differentiability.Its parameters are convolutional weights, while more complex modules would be costlier and more prone to overfitting.
  • Internal Learner: The learner minimizes a weighted squared error between target-module outputs and generated labels over a few-shot support set, with a learned regularization parameter.The support set usually contains one annotated frame but can include additional frames.
  • Internal Learner: Because the objective is a convex quadratic but direct solutions are computationally impractical, the learner uses differentiable steepest-descent iterations.The iterations exploit the convolutional structure while approximating the optimization objective.

4 Experiments

The approach is evaluated on YouTube-VOS and DAVIS benchmarks, with state-of-the-art performance on YouTube-VOS and strong results under DAVIS and bounding-box initialization settings.

  • Experimental setup: The method runs at 6 FPS on single-object sequences and is evaluated on YouTube-VOS and DAVIS 2017.The experiments use standard VOS benchmarks, with additional analyses on sampled YouTube-VOS training videos.
  • YouTube-VOS comparison: 81.5 overall G-score on YouTube-VOS 2018 represents a relative improvement of over 2.6% over STM’s 79.4.The comparison uses the benchmark’s average over Jaccard and boundary scores for seen and unseen categories.
  • Qualitative analysis: Qualitative results show accurate segmentation under occlusions, distractors, and appearance changes, while severe occlusion and similar objects produce a failure case.The failure example is attributed to both severe occlusions and highly similar objects.
  • YouTube-VOS comparison: The approach achieves a 15.4% and 15.5% relative improvement over AGSS-VOS on the J_unseen and F_unseen scores, respectively.These results indicate performance on novel categories in the benchmark setting.
  • DAVIS comparison: On DAVIS 2017, the method is within 0.2 overall-score points of STM and exceeds FRTM by 7.4% when trained only on DAVIS data.The latter setting uses only DAVIS 2017 training data and ImageNet initialization for the backbone.
  • Bounding-box initialization: With bounding-box initialization, the method improves YouTube-VOS G-score by 2.8% over Siam-RCNN and reaches 70.6 J &F on DAVIS 2017.The paper reports that the approach outperforms several mask-initialized methods on YouTube-VOS and is on par with Siam-RCNN on DAVIS.

5 Conclusions

The paper concludes that a differentiable optimization-based few-shot learner can be integrated into an end-to-end VOS architecture, while learning target labels and importance weights to produce richer target representations.

  • Conclusions: The proposed VOS approach integrates an optimization-based few-shot learner into a differentiable, end-to-end trainable architecture.The architecture is trained to maximize segmentation accuracy on annotated videos.
  • Conclusions: Learned target labels and importance weights guide the few-shot objective and enable the target model to predict a rich representation for segmentation.The rich target representation guides the VOS network in generating accurate segmentation masks.

A Derivation of Internal Learner Iteration Steps

The internal learner uses steepest-descent updates to minimize its loss, with gradients and step lengths derived in matrix form so they can be computed through neural-network operations.

  • Convolution is expressed as matrix multiplication after vectorizing the feature map and filter weights.The representation uses vec(x_t ∗ τ) = X_t τ̄.
  • The loss uses vectorized label encodings and a diagonal matrix representing point-wise importance weighting.The weighted residual is written using Eθ(y_t) and Wθ(y_t).
  • The learner updates parameters along the gradient direction using a step length selected to minimize the current loss.The update is τ̄_i+1 = τ̄_i − α_i ḡ_i.
  • The gradient is computed by the chain rule, with transposed convolution corresponding to multiplication by the transposed convolution matrix.This enables the required learner operations to be implemented with neural-network primitives.
  • Because the loss is convex, the optimal step length is obtained by solving the stationary-point condition.The derivation sets the derivative with respect to the step length to zero.

B.1 Few-shot Label Generator Eθ and Weight Predictor Wθ

The label generator and importance weight predictor share a mask feature extractor, then produce learned labels and weights from its 64-channel representation.

  • Both modules process the ground-truth segmentation mask through a convolutional layer and two residual blocks.The shared feature extractor produces a deep mask representation.
  • The importance weight predictor Wθ uses a convolutional layer to predict importance weights from the same mask features.Unlike Eθ, the described Wθ module has no ReLU activation.
  • The shared mask representation contains 64 channels and has one-sixteenth the input mask’s spatial resolution.This lower-resolution representation is used by both prediction modules.
  • The label generator Eθ uses a convolutional layer and ReLU to predict the few-shot learner’s ground-truth label.Its input is the learned representation of the ground-truth mask.

B.2 Segmentation Decoder Dθ

The segmentation decoder combines the target module’s mask encoding with backbone features in a U-Net-style hierarchy to predict the final segmentation mask.

  • The decoder takes the target module Tτ’s mask encoding together with features from the ResNet-50 backbone.These inputs provide target-specific information and image features for final prediction.
  • Its four decoder blocks correspond to the four residual blocks of the ResNet feature extractor.This establishes a multi-level decoder structure.
  • Each decoder block projects backbone features, concatenates them with the mask encoding, and processes them through convolutional and residual layers.The block also merges information from deeper and shallower decoder levels.
  • Channel attention blocks are used when merging features from deeper decoder modules.The resulting hierarchy follows the described U-Net-based design.

B.3 Bounding Box Encoder Bθ

The box encoder combines a downsampled bounding-box mask with backbone image features, while inference tracks the target locally and constrains size changes between frames.

  • B.3 Bounding Box Encoder Bθ: The box encoder receives a bounding-box mask and deep image features, aligning the mask to the feature resolution before processing them together.The mask is downsampled by a factor of 16 and concatenated with image features.
  • B.3 Bounding Box Encoder Bθ: The combined representation is processed by two residual blocks and a convolutional block that predicts the target-object mask representation.This is the architecture described for Bθ.
  • B.3 Bounding Box Encoder Bθ: The backbone features are reduced to C = 512 channels before being combined with the box representation.The reducing convolution shares weights with the target model’s projection layer.
  • B.3 Bounding Box Encoder Bθ: Inference processes a local crop around the previous target location rather than the full image.The crop is five times larger than the previous target estimate, with its maximum size limited by the image dimensions.
  • B.3 Bounding Box Encoder Bθ: The current target center and size are estimated from the predicted mask, and the size change is constrained to the range 0.95 to 1.1.The center is computed as the mask’s center of mass, while the size is estimated from its spatial spread.

D Training Details

The approach is trained on YouTube-VOS 2019 and DAVIS 2017, with longer training used for final state-of-the-art comparisons. On YouTube-VOS 2019 validation, it achieves an overall G score of 81.0, outperforming STM by 1.8 points.

  • Training procedure: Training samples come from YouTube-VOS 2019 and DAVIS 2017, with YouTube-VOS sampled six times more often.Sequences are sampled without replacement, and the network uses ADAM optimization.
  • Training procedure: Final state-of-the-art networks use a 170k-iteration long training schedule with staged learning-rate reductions and initially frozen backbone weights.The base learning rate is 10^-2, reduced after 40k, 95k, and 145k iterations; backbone weights are frozen for the first 70k iterations.
  • Training procedure: The bounding box encoder is trained on YouTube-VOS 2019 and MSCOCO for 50k iterations, updating only its convolutional layers.MSCOCO images are sampled with twice the probability of YouTube-VOS images.
  • Evaluation: 81.0 overall G on YouTube-VOS 2019 validation outperforms STM by +1.8 points.The validation set contains 507 sequences and reports Jaccard and boundary scores for seen and unseen categories.

F Detailed Ablative Study

Ablations show that richer target encodings, suitable initialization and loss choices, long training, local search, and online target updates each affect segmentation performance. The strongest reported trade-off is that long training improves J score by 1.4 but requires eight times more GPU hours.

  • Training choices: +1.2 J score results from initializing the backbone with Mask-RCNN weights instead of ImageNet weights.The comparison is performed on 300 YouTube-VOS 2019 training-set videos.
  • Training choices: +0.6 J score results from Lovasz loss compared with binary cross-entropy loss.The comparison evaluates alternative segmentation losses during training.
  • Training choices: +1.4 J score comes from long training versus short training, but long training requires 8 times more GPU hours.Long training also trains the backbone and uses a larger batch size.
  • Target representation: +2.0 J score results from using at least four mask-encoding channels instead of a single channel.More channels provide a richer representation of the target mask.
  • Inference updates: A single steepest-descent update per frame improves J score by more than 5 points over no target-model updates.Three update iterations give the best reported performance at J = 81.2, while more than three causes slight degradation from overfitting.
  • Evaluation mode: +1.0 J score results from operating on a local search region rather than the full image.The search region is based on the estimated target mask from the previous frame.

G Qualitative Results

Qualitative results compare the approach with STM under appearance changes, occlusions, and distractors, while visualizations examine learned mask encodings and box-based initialization. The comparisons show both methods have sequence-specific strengths and failures.

  • DAVIS comparison: On DAVIS sequences with occlusions, appearance changes, and distractors, the approach segments some targets STM misses but also has sequence-specific failures.It segments the gun in a late shooter frame and avoids major false predictions on soapbox, but struggles with the box in loading and one piglet in pigs.
  • Learned mask encodings: Learned label channels represent target-mask boundaries, background, and low-resolution mask structure, with consistently higher importance near target edges.Importance weights can be higher or lower on background versus foreground depending on the channel.
  • Bounding-box initialization: With bounding-box initialization, the decoder predicts a first-frame mask that serves as pseudo-ground truth for subsequent VOS processing.The predicted initial mask is less accurate than ground truth, yet subsequent frames can receive high-quality segmentation masks.
Loading 2003.11540v2…