Source-linked AI summary
Weakly Supervised Semantic Segmentation using Web-Crawled Videos
Seunghoon Hong, Donghun Yeo, Suha Kwak, Honglak Lee, Bohyung Han
TL;DR
The paper tackles weakly supervised semantic segmentation when pixel-wise masks are unavailable and image-level supervision tends to provide incomplete object coverage. It automatically retrieves web videos and combines image localization with video cues to generate masks for training. The resulting method outperforms approaches using the same supervision and is competitive with methods using extra annotations.
Problem
Pixel-wise masks are expensive, while image-level supervision can yield incomplete segmentation annotations and existing remedies may require additional human intervention.
Method
The framework retrieves videos using target class labels, filters and localizes relevant content with an image-trained encoder, and combines attention with motion and color cues to generate segmentation masks.
Results
The method outperformed previous approaches using the same supervision and was competitive with approaches relying on extra annotations.
Takeaways & Limitations
Web-crawled videos can provide sufficiently strong automatically generated segmentation annotations for learning semantic segmentation without additional supervision.
Takeaways & Limitations
Retrieved videos carry only video-level labels, and relevant frames and regions may be ambiguous because only subsets of frames or moving regions correspond to the target class.
Abstract
from arXiv · showhide
We propose a novel algorithm for weakly supervised semantic segmentation based on image-level class labels only. In weakly supervised setting, it is commonly observed that trained model overly focuses on discriminative parts rather than the entire object area. Our goal is to overcome this limitation with no additional human intervention by retrieving videos relevant to target class labels from web repository, and generating segmentation labels from the retrieved videos to simulate strong supervision for semantic segmentation. During this process, we take advantage of image classification with discriminative localization technique to reject false alarms in retrieved videos and identify relevant spatio-temporal volumes within retrieved videos. Although the entire procedure does not require any additional supervision, the segmentation annotations obtained from videos are sufficiently strong to learn a model for semantic segmentation. The proposed algorithm substantially outperforms existing methods based on the same level of supervision and is even as competitive as the approaches relying on extra annotations.
1. Introduction
The paper addresses the cost and limited scalability of pixel-wise annotation by using automatically retrieved web videos to generate segmentation supervision without additional human intervention. It combines image-based localization with video cues to reduce retrieval and segmentation ambiguities, then learns semantic segmentation from the generated masks.
- Motivation: Pixel-wise segmentation annotation is costly, so fully supervised models remain limited to a small number of classes.Large-scale datasets require manually collected masks, which demand substantial effort and expense.
- Motivation: Image-level labels reduce annotation cost but can produce suboptimal segmentation because estimated annotations lack quality measurement.Additional supervision such as boxes or scribbles can help, but often requires human intervention or domain-specific knowledge.
- Approach: Videos provide shape and extent information, while weakly annotated images help reduce temporal and spatial ambiguities in web-crawled data.Only some video frames and moving regions may correspond to the queried class, motivating the complementary use of both domains.
- Approach: The method combines discriminative localization from images with motion and color cues to generate spatio-temporal object segments from videos.An image-trained encoder filters irrelevant frames and identifies discriminative regions before graph-based segmentation produces object candidates.
- Approach: The framework automatically retrieves target-relevant videos from web repositories, avoiding human intervention for extra supervision.Retrieved videos receive weak video-level labels from their query keywords.
- Results: The framework substantially outperforms prior weakly supervised semantic segmentation methods on PASCAL VOC.The paper reports effectiveness on the benchmark without requiring additional human intervention.
2. Related Work
Related work reduces segmentation annotation costs through weak supervision and retrieves training examples from the Web, but both settings must address missing supervision and noisy data. Image-level labels are inexpensive yet remain substantially behind fully supervised methods without additional annotations.
- Semantic Segmentation: DCNN-based semantic segmentation requires expensive pixel-wise annotations, restricting prior studies largely to small-scale datasets.Fully convolutional architectures and related improvements depend on dense ground-truth masks.
- Weak Supervision: Image-level labels minimize annotation cost, but weakly supervised results remain far behind fully supervised methods because segmentation supervision is missing.Point, scribble, bounding-box, or cross-class mask annotations can narrow the gap but increase annotation requirements.
- Webly-Supervised Learning: Webly-supervised learning retrieves training examples from online resources for tasks including recognition, localization, and fine-grained categorization.Its main challenge is learning from noisy web data.
3. Our Framework
The framework combines an image-trained encoder with web-crawled videos to generate segmentation annotations, then trains a decoder for semantic segmentation on still images. It filters irrelevant video content, localizes target objects, and refines masks using spatio-temporal cues.
- Architecture: The decoupled encoder-decoder architecture uses an encoder for image classification and localization and a decoder for dense segmentation.The two components are trained separately with different data.
- Learning to Attend from Images: Weakly annotated images train the encoder to produce class predictions and coarse attention maps identifying discriminative regions.The encoder is trained under a classification objective and uses CAM to highlight class-relevant image regions.
- Generating Segmentation from Videos: The learned encoder filters irrelevant frames from web-crawled videos and identifies relevant intervals for target-class segmentation.Frames are selected using encoder-based class-relevance scores, with relevant videos formed from sufficiently long consecutive frame runs.
- Generating Segmentation from Videos: Video object masks are inferred by graph-based optimization over superpixels using attention, appearance, motion, and spatial-temporal smoothness cues.The unary term combines attention, appearance, and motion, while the pairwise term imposes spatial and temporal smoothness.
- Learning to Segment from Videos: Generated video segmentations supervise decoder training, after which the complete encoder-decoder network segments still images.The obtained segmentations serve as annotations for learning the decoder.
- Generating Segmentation from Videos: Multi-scale attention maps are merged per frame to capture object parts and shapes better than single-scale attention.The maps are resized to the original frame size and combined through max-pooling over scale.
4. Video Retrieval from Web Repository
The method retrieves class-specific videos from YouTube, then filters noisy results using thumbnails and key-frames before extracting informative video intervals.
- Videos are collected from YouTube using each predefined semantic class label as a search keyword, but retrieval is generally noisy.The noise arises because videos often lack side-information important for text-based search.
- Thumbnails and key-frames provide global and local video summaries for efficiently filtering irrelevant examples without processing whole videos.The strategy downloads thumbnails instead of entire videos and evaluates them with the image-trained encoder.
- Thumbnail classification scores identify videos likely to contain informative frames relevant to the target class.
- Reference frames used for video compression serve as key-frames, enabling informative interval selection without decompressing the entire video.
5. Experiments
Experiments on PASCAL VOC and YouTube-object evaluate the framework’s components, weakly supervised segmentation performance, and video-mask quality. Results attribute gains to separate training, automatically collected videos, domain adaptation, and semantically informed localization.
- Experimental setup: 10,582 PASCAL VOC 2012 training images and 4,606 videos with 960,517 frames form the experimental data; mIoU is the segmentation metric.The video collection uses up to 300 videos and 15 key-frames per video, with a classification threshold of 0.8.
- Internal analysis: Separate training substantially outperforms learning a whole model directly from weakly annotated videos under identical YouTube-object conditions.The comparison uses the same manually collected videos for 10 PASCAL object classes.
- Internal analysis: 6% mIoU improvement follows replacing manually collected videos with automatically collected videos, supporting larger data and informative retrieval from noisy repositories.The authors attribute the gain to better object-shape learning and effective video collection.
- Internal analysis: 3% mIoU improvement follows domain adaptation from videos to still images, where generated image annotations help learn multi-class context and different data distributions.The model is applied to weakly annotated images and fine-tuned using generated segmentation annotations.
- Comparisons to other methods: More than 7% mIoU improvement over the state-of-the-art image-level-label result is reported, with performance competitive with methods using extra supervision.The method also surpasses some approaches using point supervision or masks from other classes while requiring only image-level labels for human supervision.
- Video segmentation results: Video segmentation substantially outperforms methods based on low-level cues and some bounding-box-trained detectors, while qualitative results show more accurate object locations, boundaries, and masks under video challenges.The authors associate this with semantically meaningful attention maps and finer-grained localization than coarse detector boxes.
6. Conclusion
The framework uses automatically retrieved videos and generated object masks to provide segmentation supervision from image-level labels alone. It outperforms methods with the same supervision and is competitive with approaches using extra supervision.
- 6. Conclusion: The framework automatically retrieves relevant Web videos and generates object masks to simulate segmentation supervision without additional human intervention.An encoder predicts attention maps, which are combined with motion cues to estimate object shape and extent before training a segmentation decoder.
- 6. Conclusion: The resulting model outperforms previous approaches using the same supervision level and is competitive with methods relying on extra supervision.The conclusion attributes the approach’s supervision source to image-level class labels only.