Source-linked AI summary
Few-Shot Segmentation Propagation with Guided Networks
Kate Rakelly, Evan Shelhamer, Trevor Darrell, Alexei A. Efros, Sergey Levine
TL;DR
Dense segmentation labels and task-specific methods make it difficult to apply supervision across inputs or correct inference with additional guidance. Guided networks extract a latent task representation from sparse annotations and propagate it to query images; the paper reports state-of-the-art accuracy for the annotation and time required across evaluated segmentation tasks.
Problem
Segmentation often requires costly dense annotations, while existing methods are task-specific and interactive annotations cannot guide segmentation of new inputs.
Source: §1 · p. 1Method
Results
The method achieves state-of-the-art accuracy for the annotation and time required across interactive image, semantic, and video segmentation tasks.
Source: §1 · p. 2Takeaways & Limitations
Guided segmentation can switch tasks, incorporate further annotations near-instantly, and segment new inputs without the supervisor.
Source: §6 · p. 9For semantic segmentation, one-shot support may fail to cover within-class visual variation, limiting improvement from additional shots.
Source: §5.2 · p. 8
Abstract
from arXiv · showhide
Learning-based methods for visual segmentation have made progress on particular types of segmentation tasks, but are limited by the necessary supervision, the narrow definitions of fixed tasks, and the lack of control during inference for correcting errors. To remedy the rigidity and annotation burden of standard approaches, we address the problem of few-shot segmentation: given few image and few pixel supervision, segment any images accordingly. We propose guided networks, which extract a latent task representation from any amount of supervision, and optimize our architecture end-to-end for fast, accurate few-shot segmentation. Our method can switch tasks without further optimization and quickly update when given more guidance. We report the first results for segmentation from one pixel per concept and show real-time interactive video segmentation. Our unified approach propagates pixel annotations across space for interactive segmentation, across time for video segmentation, and across scenes for semantic segmentation. Our guided segmentor is state-of-the-art in accuracy for the amount of annotation and time. See http://github.com/shelhamer/revolver for code, models, and more details.
1 Introduction
The introduction frames segmentation as costly to annotate, task-specific, and difficult to correct, then proposes few-shot segmentation with guided networks to propagate sparse pixel supervision to new inputs. The method is reported to achieve state-of-the-art accuracy for the annotation and time required across several segmentation tasks.
1 Introduction
- Dense segmentation supervision is costly and time-consuming to collect, while existing propagation and interactive methods are limited in task scope or in guiding new inputs.The introduction notes that dense datasets may contain billions of pixelwise annotations and that image-specific interactive annotations cannot inform segmentation of another input.Source: §1 · p. 1
- The method addresses few-shot segmentation by using sparse pixelwise annotations to define a task and segment unannotated images accordingly.The paper reports segmentation from as little as one pixel per concept and updates with further annotations.Source: §1 · pp. 1–2
- Guided networks extract a latent task representation from annotated support and use it to segment query images, without further optimization for new tasks.Additional annotations can be incorporated incrementally to alter the task or correct errors.Source: §1 · p. 2
- Across interactive image, semantic, video object, and real-time interactive video segmentation, accuracy is state-of-the-art for the annotation and time required.The speed of incorporating annotations makes the method suitable for real-time interactive use.Source: §1 · p. 2
2 Related Work
The related-work section contrasts few-shot segmentation with task-specific segmentation and classification methods, emphasizing sparse pixel supervision, structured outputs, and propagation across images. It positions guided networks as a framework connecting these areas.
Segmentation
- The framework connects segmentation with few-shot learning and treats semantic, interactive, and video object segmentation as challenge problems.Figure 2 summarizes these segmentation problem types.Source: §2 · p. 2, Figure 2 · p. 3
- OSLSM requires dense annotations and semantic training supervision, OSVOS fine-tunes at inference and struggles with sparse labels, and DIOS cannot propagate annotations across images.The proposed approach instead learns feedforward guidance from sparse annotations and can segment new inputs without requiring annotations on each one.Source: §2 · p. 2
Few-shot learning
- Prior few-shot methods largely target classification, while this work addresses segmentation across extremely sparse to dense annotations and large-scale visual recognition.The section highlights sparse and imbalanced supervision as underexplored for structured output.Source: §2 · p. 3
- Existing few-shot approaches use inference-time optimization, embedding-based retrieval, or support-conditioned parameter modulation, each with stated limits for visual structured output.The section identifies computational impracticality, degradation at higher shot and way, and difficulty merging modulations for multiple shots.Source: §2 · p. 3
3 Few-Shot Segmentation
Few-shot segmentation defines a task through annotated support images and unannotated query images, with pixel-level supervision extending the usual few-shot setting. The approach extracts a task representation from support and conditions pixelwise query segmentation on it.
3 Few-Shot Segmentation
- Sparse annotations allow few-shot tasks to be specified by support images and annotated pixels per image, with (S, P)-shot settings capturing both quantities.The paper emphasizes small P as more practical to collect and also considers mixed-shot tasks with varying annotation amounts.Source: §3 · p. 3
- A task consists of annotated support and unannotated query images, with point-label support annotations and query segmentation targets defined over query pixels.The formulation denotes support count by S, query count by Q, and labels as one of K classes or unknown.Source: §3 · p. 3
- Tasks are treated as binary positive-versus-negative segmentation, while higher-way tasks can be represented as a union of binary tasks.The paper sets Q = 1 because query inference is independent in its model.Source: §3 · pp. 3–4
- Figure 3 contrasts early fusion by concatenation with late fusion into image and annotation streams, and shows global guidance propagating appearance across locations.In the example, annotating one bird guides segmentation of similar-looking birds elsewhere.Source: Figure 3 · p. 4
- The method has two stages: extract a task representation from structured support, then segment the query conditioned on that representation.The task representation and encoder must handle image and pixel hierarchy, variable dimensions, unannotated support pixels, and skewed distributions.Source: §3 · p. 4
4 Guided Networks
Guided networks extract a task representation from annotated support and use it to condition query segmentation. The guide and inference branches are jointly optimized through episodic training for structured output.
4 Guided Networks
- The guide extracts latent task representation z from annotated support, and the inference branch uses z to segment query images.Source: §4 · p. 4
4.1 Guidance: From Support to Task Representation
- Late fusion combines image features with annotation masks through element-wise multiplication, allowing image features to be cached when guidance changes.Source: §4.1 · p. 5
- For different support and query images, global pooling discards spatial location; for the same image, identity pooling can retain informative feature locations.Source: §4.1 · p. 5
- Multi-shot support merges shot-wise image-annotation representations, including when positive and negative annotations occur in different images.Source: §4.1 · p. 5
4.2 Guiding Inference
- Feature fusion was more accurate and simpler to tune than parameter regression and prototypes, so it was used for all reported results.Source: §4.2 · p. 6
4.3 Optimization and Task Sampling
- Episodic training samples a task and support/query images from dense labels, then jointly optimizes the guide and segmentor against dense query targets.Source: §4.3 · pp. 6–7
5 Results
The section evaluates interactive, semantic, and video segmentation as few-shot tasks, then reports benchmark results and a simulated-dot interactive video evaluation.
5 Results
- The evaluation frames interactive, semantic, and video segmentation as binary support-versus-complement tasks, using positive IU and fine-tuning and foreground-background baselines.IU is selected as a common metric across tasks.Source: §5 · p. 7
5.1 Interactive Image Segmentation
- For interactive image selection, treating support and query images as identical makes the task a special case of few-shot segmentation; Revolver is more accurate under extreme sparsity and faster to update than DIOS.DIOS uses early fusion, while Revolver uses late-global fusion; this result motivates using late-global guidance throughout.Source: §5.1 · p. 7
5.4 Interactive Video Segmentation
- Figure 7 pairs sparse/dense video accuracy-time evaluation with simulated-dot interactive video results; accuracy improves with more image or pixel guidance.The guided architecture is feedforward and fast, and updates faster still when annotations change.Source: Figure 7 · p. 8, §5.4 · p. 9
5.2 Few-shot Semantic Segmentation
- Revolver achieves state-of-the-art sparse semantic segmentation results that rival the most accurate dense results with just two labeled pixels.OSLSM is defined only for {0, 1} annotations, so missing annotations are mapped to negative for its evaluation.Source: §5.2 · p. 8
- Class-supervised guidance is insensitive to annotation amount because its task representation has exceptionally low variance across supports from the same class.One support image may not cover within-class variation, such as differences in color, texture, or pose.Source: §5.2 · p. 8
- With one positive/negative semantic pixel, instance-trained guidance reaches 26.4% accuracy, exceeding OSLSM; its accuracy improves by +6 at 10 pixels while class-supervised accuracy is near constant.The instance-trained model was not trained on classes.Source: §5.2 · p. 8
5.3 Video Object Segmentation
- 33.3% accuracy yields 80% relative improvement over methods in the same time envelope; five positive and negative points retain 87% of dense accuracy.Guidance takes 3 sec/video, about 200× faster than fine-tuning, which takes 10+min/video.Source: §5.3 · p. 8
6 Discussion
Few-shot segmentation unifies annotation-bound tasks through guided networks that extract a latent task representation from supervision. This lets the segmentor switch tasks, improve accuracy near-instantly with added annotations, and segment new inputs without a supervisor.
6 Discussion
- Few-shot segmentation unifies annotation-bound segmentation problems.Guided networks reconcile task-driven and interactive inference.Source: §6 · p. 9
- Guided networks extract a latent task representation from any amount of supervision.The resulting guidance directs segmentation across tasks.Source: §6 · p. 9
- With guidance, the segmentor can switch tasks, improve accuracy near-instantly with more annotations, and segment new inputs without a supervisor.Source: §6 · p. 9