Source-linked AI summary
Learning to Refine Object Segments
Pedro O. Pinheiro, Tsung-Yi Lin, Ronan Collobert, Piotr Dollàr
TL;DR
Object segmentation must combine high-level object information with low-level spatial detail, which standard feedforward CNNs represent unevenly. SharpMask adds top-down refinement to a DeepMask-based architecture, improving average recall by 10–20% while reducing runtime by 50%.
Problem
Feedforward CNNs provide semantic object information but lose pixel-level resolution, limiting accurate object-boundary labeling in instance segmentation.
Method
SharpMask generates a coarse mask encoding feedforward, then successively refines it by merging top-down representations with matching lower-level features.
Results
10–20% higher average recall on COCO object proposal generation and 50% faster speed than DeepMask, averaging 0.76s per image.
Takeaways & Limitations
SharpMask establishes state-of-the-art object proposal performance and speed while producing higher-fidelity masks with sharper object boundaries.
Takeaways & Limitations
Reducing spatial resolution hurts instance-segmentation performance, while preserving resolution at greater depth would require training networks from scratch, which the authors leave for future work.
Abstract
from arXiv · showhide
Object segmentation requires both object-level information and low-level pixel data. This presents a challenge for feedforward networks: lower layers in convolutional nets capture rich spatial information, while upper layers encode object-level knowledge but are invariant to factors such as pose and appearance. In this work we propose to augment feedforward nets for object segmentation with a novel top-down refinement approach. The resulting bottom-up/top-down architecture is capable of efficiently generating high-fidelity object masks. Similarly to skip connections, our approach leverages features at all layers of the net. Unlike skip connections, our approach does not attempt to output independent predictions at each layer. Instead, we first output a coarse `mask encoding' in a feedforward pass, then refine this mask encoding in a top-down pass utilizing features at successively lower layers. The approach is simple, fast, and effective. Building on the recent DeepMask network for generating object proposals, we show accuracy improvements of 10-20% in average recall for various setups. Additionally, by optimizing the overall network architecture, our approach, which we call SharpMask, is 50% faster than the original DeepMask network (under .8s per image).
1 Introduction
SharpMask addresses the tension between high-level object knowledge and low-level spatial detail by refining a feedforward mask encoding with progressively earlier features. Built on DeepMask, it improves COCO proposal recall and speed while producing sharper masks.
- Motivation: Object instance segmentation requires both identifying each object instance and producing accurate pixel-level masks, combining challenges from detection and semantic segmentation [9].Standard feedforward CNNs produce coarse, highly semantic representations through pooling, which limits pixel-accurate mask generation.
- Method: Unlike skip architectures that independently predict from multiple layers and average the results, SharpMask refines one evolving mask representation.This design targets object instance segmentation, where local receptive fields alone are insufficient to capture object-level structure.
- Method: SharpMask combines low-level spatial information with upper-layer object knowledge by refining a coarse feedforward mask encoding through successive top-down stages.Each refinement module merges the top-down mask encoding with matching bottom-up features, doubles spatial resolution, and remains fully backpropable.
- Method: The method augments DeepMask [22] with top-down refinement and also optimizes DeepMask’s bottom-up architecture to improve segmentation quality and efficiency.The resulting approach is named SharpMask for its sharper, higher-fidelity object masks.
- Results: 10–20% higher average recall on COCO establishes SharpMask as state of the art for object proposal generation, while its optimized model is 50% faster than DeepMask at .76s per image.The fast model runs at .46s and additional image scales can improve small-object recall by ∼2×.
2 Related Work
Prior CNN-based pixel-labeling methods include multiscale, deconvolutional, and graphical-model architectures, but severe computational constraints limit their applicability to refining hundreds of object proposals per image. SharpMask instead achieves efficient refinement without recomputing features at multiple image scales, while transferring richer feature information than deconvolutional switches and avoiding graphical-model runtimes.
- CNNs have been successfully applied to depth estimation, optical flow, and semantic segmentation, motivating architectural innovations for pixel labeling.
- Most prior architectures are unsuitable for object proposals because refining hundreds of proposals per image requires minimal marginal computation per proposal.
- Multiscale architectures: Multiscale methods process multiple image resolutions, whereas SharpMask uses similar coarse-to-fine intuition without recomputing features at each scale, enabling efficient refinement across many locations.
- Deconvolutional networks and graphical model networks: Deconvolutional networks and graphical-model CNNs [18] sharpen coarse masks, but SharpMask transfers feature values rather than limiting communication to pooling switches and avoids their proposal-level computational cost.
3 Learning Mask Refinement
SharpMask augments DeepMask with a top-down refinement pathway that combines semantic mask encodings with spatially detailed features, producing sharper, pixel-accurate object masks. The method progressively upsamples mask encodings through learned refinement modules while retaining efficient, fully backpropagable computation.
- Motivation: SharpMask addresses DeepMask’s coarse boundary alignment by augmenting it with refinement that produces sharper, pixel-accurate object masks.DeepMask captures object-level shape but aligns coarsely with boundaries; SharpMask improves boundary quality.
- Architecture: The architecture merges high-level semantic information with low-level spatial detail through a feedforward mask encoding followed by top-down refinement.The design uses object-level information first, then successively integrates earlier-layer features while reversing pooling-induced resolution loss.
- Architecture: Each refinement module combines a coarse mask encoding M_i with matching bottom-up features F_i, doubles spatial resolution, and iteratively produces full-resolution pixel labels.The learned operation is M_i+1 = R_i(M_i, F_i), with one module per pooling layer.
- Mask Encoding: The feedforward pathway creates a low-resolution semantic mask encoding whose multiple channels capture more information than a simple segmentation mask.Using k_1^m > 1 is identified as important for achieving good accuracy.
- Refinement Module: To control computation, each module compresses bottom-up features into skip features, concatenates them with the mask encoding, applies convolution and ReLU, and bilinearly upsamples by 2.The resulting module uses only convolution, ReLU, bilinear upsampling, and concatenation, making it fully backpropagable and efficient.
- Training: SharpMask uses the same data definition and loss as DeepMask, initializes its trunk from ImageNet pretraining, and initializes remaining layers randomly.Training samples contain an input patch, an object-presence-and-scale label, and a binary mask for positive samples.
4 Feedforward Architecture
The feedforward architecture study identifies computational bottlenecks in DeepMask’s trunk and head, then explores design tradeoffs affecting segmentation accuracy and inference speed. Stride density is especially important for mask prediction, while simplified heads and dimensionality reduction improve efficiency.
- Trunk: DeepMask’s feature extraction, mask prediction, and score prediction consume 40%, 40%, and 20% of runtime, making deeper or broader trunks costly.Replacing the 11-layer VGG-A trunk with 16-layer VGG-D can double runtime; the authors use a 50-layer Residual Network.
- Trunk: Reducing input size or increasing pooling speeds inference but lowers resolution and mask accuracy, while increasing depth can hurt performance through reduced spatial resolution.The authors examine input size W, pooling layers P, stride density S, depth D, and feature channels F; training deeper networks without reducing resolution is left for future work.
- Trunk: Stride density is key for mask prediction: doubling stride at fixed input size greatly reduces performance by requiring greater spatial invariance.Stride density is defined as S=W/stride, and denser overlap with ground-truth locations improves mask prediction.
- Trunk: A 1×1 convolution reduces high-dimensional top-layer channels before feature aggregation, producing substantial speedups.High-dimensional features create an aggregation bottleneck, motivating dimensionality reduction before aggregation.
- Head: The head simplifies DeepMask’s slow mask-and-score branching through shared computation: Head A removes interleaving, Head B shares features, and Head C simplifies further.DeepMask requires large convolutions for both branches, plus score-branch pooling and interleaving; the variants retain identical mask branches while progressively simplifying score computation.
5 Experiments
Experiments on COCO evaluate mask and proposal quality across IoU-based average recall, object scales, speed, and accuracy. SharpMask’s optimized feedforward base and top-down refinement together deliver faster, more accurate, higher-fidelity proposals than prior systems.
- Evaluation Protocol: COCO evaluation measures IoU-based average recall at 10, 100, and 1000 proposals, summarizes these counts with AUC, and reports results separately for small, medium, and large objects.Most experiments use the first 5k validation images, while a separate validation subset selects architectures and hyperparameters.
- Feedforward Architecture: DeepMask-ours is over 3× faster than DeepMask, reducing per-image time from 1.59s to .46s while improving accuracy and shrinking parameters from ∼75M to ∼17M.The architecture combines the W160-P4-D39-F128 trunk with head C and is used as SharpMask’s feedforward base.
- Comparison with Existing Methods: SharpMask achieves state-of-the-art speed and accuracy across all COCO box- and segmentation-proposal metrics by a large margin.Table 3 compares SharpMask with existing methods on both box and segmentation proposals, deriving boxes from tight bounds around predicted masks.
- Top-Down Refinement: Top-down refinement produces a considerable AR boost over the identical-score-branch DeepMask-ours baseline, with larger gains for segmentation than box predictions.The shared score branch isolates improvements attributable to refinement, while sharpening masks need not substantially change tight object boxes.
- Detection Results: 28 AP for SharpMask is 5 AP higher than SelSearch on COCO bounding-box detection, with performance converging at approximately 500 proposals per image.Both methods use the MPN classifier in the Fig. 5c comparison.
6 Conclusion
The paper introduces a feedforward architecture augmented with top-down refinement modules for object instance segmentation. It achieves state-of-the-art object proposal generation in performance and speed, while the refinement approach can extend to other pixel-labeling tasks.
- The proposed architecture augments feedforward networks with top-down refinement modules for object instance segmentation.
- The model achieves a new state of the art for object proposal generation in both performance and speed.
- The refinement approach is general and could be applied to other pixel-labeling tasks.