Source-linked AI summary
Pixel-Level Matching for Video Object Segmentation using Convolutional Neural Networks
Jae Shin Yoon, Francois Rameau, Junsik Kim, Seokju Lee, Seunghak Shin, In So Kweon
TL;DR
Video object segmentation must propagate an initial mask across frames despite clutter, appearance changes, and undefined target classes. This paper uses CNN-based pixel-level matching with multi-level features, compression, and two-stage training; experiments report better accuracy, speed, and stability than related methods, alongside transferability to other domains.
Problem
Video object segmentation remains difficult because existing methods can be computationally expensive, drift in cluttered backgrounds, and struggle with scale, appearance, and object-class variation.
Method
The method matches target and background pixels with a Siamese CNN using multi-level features, compressed representations, and pre-training followed by fine-tuning.
Results
Experiments on three benchmark datasets report better accuracy, speed, and stability than previous related methods, with transferability to different domains also demonstrated.
Takeaways & Limitations
The network maintains object representation capability while reducing memory requirements and can be transferred from RGB data to thermal infrared data through fine-tuning.
Takeaways & Limitations
Hard-negative data without any target-object part proved rather inefficient, and model updates are avoided because imperfect intermediate labels can increase drift.
Abstract
from arXiv · showhide
We propose a novel video object segmentation algorithm based on pixel-level matching using Convolutional Neural Networks (CNN). Our network aims to distinguish the target area from the background on the basis of the pixel-level similarity between two object units. The proposed network represents a target object using features from different depth layers in order to take advantage of both the spatial details and the category-level semantic information. Furthermore, we propose a feature compression technique that drastically reduces the memory requirements while maintaining the capability of feature representation. Two-stage training (pre-training and fine-tuning) allows our network to handle any target object regardless of its category (even if the object's type does not belong to the pre-training data) or of variations in its appearance through a video sequence. Experiments on large datasets demonstrate the effectiveness of our model - against related methods - in terms of accuracy, speed, and stability. Finally, we introduce the transferability of our network to different domains, such as the infrared data domain.
1. Introduction & Related Works
Video object segmentation propagates an initial mask across a video, but existing methods remain computationally costly and vulnerable to clutter, scale changes, and appearance variation. The paper addresses these issues with CNN-based pixel-level matching, multi-level features, compression, and two-stage training.
- Problem: Video object segmentation propagates initial object masks through every frame from single key-frame supervision.The task supports video summarization, stabilization, retrieval, and scene understanding.
- Problem: Graph-based approaches can be computationally expensive for long videos and suffer from clutter-induced drift, scale variation, and dynamic appearance changes.These limitations remain partly unsolved in recent methods.
- Problem: CNN methods offer semantic object representations, but video object segmentation requires training across undefined object classes and changing appearances.Training only on the initial frame can over-fit to its specific appearance.
- Approach: The proposed network performs pixel-level matching using features from lower to higher layers, combining spatial details with semantic information.A Siamese feature extractor supports matching between non-successive frames containing target appearance variations.
- Approach: Feature compression reduces network memory requirements while maintaining representation ability, and two-stage training adapts to arbitrary target classes and appearance variations.The paper also evaluates transfer from RGB pre-training to infrared data.
2. Proposed Method
The proposed method uses pixel-level matching between reference and search inputs, combining multi-level features with compression and two-stage training for robust video object segmentation. It decodes coherent objectness maps and aggregates multiple candidate responses to extract the target area.
- Training: Two-stage training combines offline pre-training on 300,000 image pairs with online fine-tuning on the first frame of a new sequence.Pre-training addresses appearance variation, while fine-tuning adapts the model to the new target object.
- Network inputs: The Siamese network receives reference and search inputs sampled from non-successive frames to improve robustness to dynamic appearance variations.The two streams share convolutional and compressor weights.
- Pixel-level matching: Multi-level features combine lower-layer spatial details with higher-layer semantic information, while three hidden FC layers encode global pixel-level similarity.The final FC representation is reshaped into a 50×50 matching score table.
- Pixel-level matching: Compressors reduce each feature output by a data compression ratio of 16 before the initial FC layer to lower memory requirements while retaining reliable information.Each compressor uses 3×3 convolution, ReLU, and local response normalization, with shared weights across the two streams.
- Objectness decoding: Convolutional decoding enforces object coherency and rejects outliers before classifying each pixel as background or target.Zero-padded 3×3 convolutions maintain spatial resolution, and the loss uses element-wise Euclidean distance with a 50×50 output.
- Video object segmentation: The online segmentation pipeline restores, aligns, and aggregates probability maps from candidate boxes before thresholding the combined response.Nine candidate samples are used as a compromise between speed and accuracy.
3. Experiments
Experiments evaluate the network’s architecture, computational behavior, stability, comparative performance, and transfer to infrared data. Results support multilayer features, normalization, compression, independent frame matching, and single-frame domain transfer.
- Comparative Evaluation: The network is evaluated on three benchmarks, including DAVIS, SegTrack, and JumpCut, with an additional infrared-domain tracking experiment.The evaluation covers accuracy, speed, stability, and transferability.
- Efficiency and Stability: 8 × 10^-3s feed-forward time and approximately 1.5×10^-1s per frame quantify the network’s computational efficiency on DAVIS.The reported metrics distinguish feed-forward processing from the entire per-frame process.
- Proposed Network Validation: Normalization produces stable L2-loss convergence without sigmoid activation, whereas loss severely fluctuates without normalization.The normalized case behaves similarly to sigmoid activation plus L2 loss.
- Proposed Network Validation: Compressors are indispensable for training: memory exceeded 12GB without them but was about 3.7GB with batch size 32.The uncompressed model exceeded the available GPU capacity even with batch size 1.
- Proposed Network Validation: Multilayer features outperform single-layer features for pixel-wise localization, while single-layer PLMS is vulnerable to cluttered-background drift.The multilayer design combines spatial details with semantic information.
- Comparative Evaluation: On DAVIS, the proposed method generally performs better than the compared approaches and remains more efficient and stable than OFL.Its frame-independent matching strategy limits the propagation of drift from one frame to subsequent frames.
4. Conclusion
The paper presents a deep learning video object segmentation method that combines pixel-level matching, two-stage training, multilayer features, and compression. Across three benchmark datasets, it reports improved accuracy, speed, and stability, while also demonstrating transfer to thermal infrared data.
- The proposed network uses encoding and decoding models for pixel-level object matching in video object segmentation.
- Two-stage training handles appearance variations while preventing over-fitting to the initial frame.
- Experiments on three widely used benchmark datasets show better accuracy, speed, and stability than previous related methods.
- Multilayer features and feature compression make the network compact while maintaining its object representation capability.
- The network transfers to different domains, demonstrated using a thermal infrared database.