Source-linked AI summary
Deep Snake for Real-Time Instance Segmentation
Sida Peng, Wen Jiang, Huaijin Pi, Xiuli Li, Hujun Bao, Xiaowei Zhou
TL;DR
Instance segmentation needs accurate, efficient representations that are less dependent on imperfect detection boxes and costly pixel-mask processing. Deep snake learns circularly structured contour deformation within a two-stage proposal-and-refinement pipeline. It achieves competitive results across four datasets and runs at 32.3 fps for 512 × 512 images on a GTX 1080ti GPU.
Problem
Pixel-wise instance segmentation can be sensitive to inaccurate bounding boxes and costly post-processing, while direct contour-coordinate regression often underperforms pixel-based methods.
Method
Deep snake uses a neural network to iteratively deform an initial contour with vertex-wise offsets, using circular convolution to learn features on the contour’s cycle graph.
Results
32.3 fps is achieved for 512 × 512 images on a GTX 1080ti GPU, with competitive performance on Cityscapes, KINS, SBD and COCO.
Takeaways & Limitations
The contour-based pipeline provides an efficient real-time instance-segmentation approach that handles localization errors and avoids costly pixel-mask post-processing.
Abstract
from arXiv · showhide
This paper introduces a novel contour-based approach named deep snake for real-time instance segmentation. Unlike some recent methods that directly regress the coordinates of the object boundary points from an image, deep snake uses a neural network to iteratively deform an initial contour to match the object boundary, which implements the classic idea of snake algorithms with a learning-based approach. For structured feature learning on the contour, we propose to use circular convolution in deep snake, which better exploits the cycle-graph structure of a contour compared against generic graph convolution. Based on deep snake, we develop a two-stage pipeline for instance segmentation: initial contour proposal and contour deformation, which can handle errors in object localization. Experiments show that the proposed approach achieves competitive performances on the Cityscapes, KINS, SBD and COCO datasets while being efficient for real-time applications with a speed of 32.3 fps for 512$\times$512 images on a 1080Ti GPU. The code is available at https://github.com/zju3dv/snake/.
1. Introduction
Deep Snake frames instance segmentation as learned contour deformation, replacing dense pixel masks or direct boundary regression with a contour-based, two-stage pipeline designed for localization errors and real-time use.
- Motivation: Pixel-wise masks can be sensitive to inaccurate detection boxes and require costly post-processing, motivating more efficient shape representations.The paper identifies object contours as a lower-parameter alternative to dense binary pixels.
- Deep snake: Deep snake learns to iteratively deform an initial contour by regressing vertex-wise offsets toward the object boundary.This learning-based approach follows the deformation principle of classic snake algorithms rather than directly regressing all boundary coordinates.
- Deep snake: Circular convolution exploits the contour’s cycle-graph structure and provides a learnable aggregation of neighboring vertex features.Unlike generic graph convolution, its kernel models each vertex together with relationships among neighboring vertices.
- Pipeline: The instance-segmentation pipeline combines initial contour proposal with contour deformation, and both stages can handle errors in object localization.The proposal uses detector information and an octagon formed from object extreme points to initialize the contour.
- Results: 32.3 fps is achieved for 512 × 512 images on a GTX 1080ti GPU while maintaining competitive performance on Cityscapes, KINS, SBD and COCO.The contour representation uses fewer parameters and avoids costly mask post-processing such as upsampling.
2. Related work
Prior instance-segmentation methods mainly use pixel-level representations, while contour-based work deforms or predicts boundary vertices using handcrafted energies or learned networks.
- Pixel-based methods: Proposal-based pixel methods segment instances within detected regions, whereas proposal-free methods cluster pixel-level auxiliary information.These approaches rely on dense pixel representations and standard CNN or clustering pipelines.
- Contour-based methods: Traditional snake methods deform an initial contour using handcrafted, typically nonconvex energy functions.Learning-based variants were introduced to improve robustness by learning the energy function from data.
- Contour-based methods: Contour-labeling methods either predict points sequentially with recurrent networks or use graph convolution to predict vertex-wise deformation offsets.The graph-convolution approach avoids sequential inference and improves annotation speed.
3. Proposed approach
Deep snake learns to deform an initial contour into an object boundary using circular convolution for contour-structured feature learning. Its instance-segmentation pipeline proposes an octagonal contour from detector outputs, then iteratively refines it, including when localization is inaccurate.
- Learning-based snake algorithm: Deep snake extracts image and coordinate features at contour vertices and predicts per-vertex offsets to deform the contour end to end.Vertex coordinates are translation-normalized by subtracting the minimum value in each dimension.
- Learning-based snake algorithm: Circular convolution preserves the closed-cycle topology of contour vertices while producing output features with the same length as the input.It extends contour features periodically before applying a standard convolution with a learnable kernel.
- Learning-based snake algorithm: Deep snake uses a backbone, fusion block, and prediction head to process a contour and output vertex-wise deformation offsets.The fusion block combines features across backbone layers and contour points.
- Deep snake for instance segmentation: The detector-to-contour pipeline shifts a diamond contour to object extreme points, constructs an octagon, and iteratively deforms it to the object boundary.The octagon is sampled into N contour points before deformation, with N set to 128 in the experiments.
- Deep snake for instance segmentation: Iterative deformation predicts offsets from the current contour, adds them to vertex coordinates, and reuses the deformed contour for the next iteration.The default number of inference iterations is 3.
- Deep snake for instance segmentation: Contour deformation can address detector localization errors by evolving the initial contour toward the object boundary.The contour provides an alternative representation of the object's spatial extent.
- Deep snake for instance segmentation: A component detector handles occluded objects split into multiple components by segmenting each component separately and merging the results.It applies RoIAlign within the detected object box before detecting component boxes.
4. Implementation details
Deep snake is trained with smooth ℓ1 losses for extreme-point prediction and iterative contour deformation, while the detector uses its original detection loss. The implementation adopts CenterNet for both object and component detection.
- Training strategy: Smooth ℓ1 loss trains the extreme-point prediction and iterative contour-deformation processes.The deformation loss compares predicted deformed contour points with ground-truth boundary points.
- Training strategy: The iterative contour-deformation loss compares each deformed contour point with its corresponding ground-truth boundary point.The ground-truth boundary is represented by sampled vertices, and the predicted point is denoted by ˜x_i.
- Training strategy: The detection component uses the same loss function as the original detection model, with dataset-specific training details described separately.The supplied implementation description does not specify additional detector-loss modifications.
- Detector: CenterNet serves as the detector for object boxes and as a class-agnostic detector for component boxes.The object detector uses class-specific boxes, while the component detector predicts component locations without class labels.
5. Experiments
Experiments evaluate deep snake across several instance-segmentation datasets and ablate its architecture, contour initialization, and convolution operator. The method achieves strong benchmark performance, with reported real-time throughput and improved contour deformation from circular convolution.
- Datasets and Metrics: The experiments cover Cityscapes, KINS, SBD, and COCO using dataset-specific instance-segmentation metrics.Cityscapes and KINS use AP, SBD reports APvol, AP50, and AP70, and COCO reports AP.
- Ablation Studies: The ablation baseline combines Curve-gcn with CenterNet, while subsequent rows add the proposed architecture, initial contour proposal, and circular convolution.The ablation studies are conducted on SBD with end-to-end training and multi-scale augmentation.
- Ablation Studies: 1.4 APvol improvement comes from replacing Graph-ResNet with the proposed architecture while retaining graph convolution.The architecture adds a global fusion block before the prediction head.
- Ablation Studies: 1.3 APvol improvement follows the addition of an octagon contour proposal based on four predicted object extreme points.The proposal is intended to compensate for detection errors and enclose objects more tightly than ellipse initialization.
- Ablation Studies: 0.8 APvol improvement follows replacing graph convolution with circular convolution, which outperforms graph convolution across tested inference iterations.Two circular-convolution iterations outperform three graph-convolution iterations by 0.6 APvol, while additional iterations do not necessarily improve performance.
- Comparison with State-of-the-Art Methods: On SBD, the method surpasses contour-based baselines by at least 19.1 APvol and improves AP50 and AP70 by 21.4 and 36.2, respectively.The comparison is against STS and ESE, which regress contour shape vectors.
- Comparison with State-of-the-Art Methods: 32.3 fps is achieved for 512×512 SBD images on a GTX 1080 Ti, while Cityscapes and KINS report state-of-the-art or best comparative performance under their tested settings.Cityscapes reports 0.9 AP over PANet on validation and 1.3 AP50 on test; KINS reports 7.6 fps at 768×2496 resolution.
6. Conclusion
The paper concludes that deep snake combines circular-convolution contour deformation with a two-stage instance-segmentation pipeline. It reports competitive results across four datasets, real-time performance, and a multi-component strategy for disconnected instances.
- Conclusion: Deep snake learns contour deformation with circular convolution and vertex-wise offset regression.The method is presented as a learning-based snake algorithm for real-time instance segmentation.
- Conclusion: The instance-segmentation pipeline consists of initial contour proposal followed by contour deformation.The paper reports that this pipeline outperforms direct regression of object-boundary coordinates.
- Conclusion: The multi-component detection strategy addresses the limitation that a single contour can outline only one connected component.Its effectiveness is demonstrated on Cityscapes.
- Conclusion: Competitive results are reported on Cityscapes, KINS, SBD, and COCO with real-time performance.The conclusion summarizes the method’s reported cross-dataset evaluation.