Source-linked AI summary
E2EC: An End-to-End Contour-based Method for High-Quality High-Speed Instance Segmentation
Tao Zhang, Shiqing Wei, Shunping Ji
TL;DR
Contour-based instance segmentation faces hand-designed initial contours and fixed predicted-label vertex pairing. E2EC combines learnable contour initialization, global deformation, multi-direction alignment, and dynamic matching loss; it reports state-of-the-art results on KINS, SBD, and Cityscapes alongside 36 fps inference for 512×512 images.
Problem
Contour-based methods use hand-designed initial contours and fixed predicted-label vertex pairings, while limited feature use also constrains contour adjustment.
Source: §1 · p. 2Method
E2EC combines learnable contour initialization and global deformation with multi-direction alignment and dynamic matching loss.
Source: §1 · p. 2Results
Takeaways & Limitations
A faster variant retaining only learnable contour initialization has accuracy comparable to Deep Snake and speed almost as fast as CenterNet's backbone one-stage detector.
Source: §6 · p. 8
Abstract
from arXiv · showhide
Contour-based instance segmentation methods have developed rapidly recently but feature rough and hand-crafted front-end contour initialization, which restricts the model performance, and an empirical and fixed backend predicted-label vertex pairing, which contributes to the learning difficulty. In this paper, we introduce a novel contour-based method, named E2EC, for high-quality instance segmentation. Firstly, E2EC applies a novel learnable contour initialization architecture instead of hand-crafted contour initialization. This consists of a contour initialization module for constructing more explicit learning goals and a global contour deformation module for taking advantage of all of the vertices' features better. Secondly, we propose a novel label sampling scheme, named multi-direction alignment, to reduce the learning difficulty. Thirdly, to improve the quality of the boundary details, we dynamically match the most appropriate predicted-ground truth vertex pairs and propose the corresponding loss function named dynamic matching loss. The experiments showed that E2EC can achieve a state-of-the-art performance on the KITTI INStance (KINS) dataset, the Semantic Boundaries Dataset (SBD), the Cityscapes and the COCO dataset. E2EC is also efficient for use in real-time applications, with an inference speed of 36 fps for 512*512 images on an NVIDIA A6000 GPU. Code will be released at https://github.com/zhang-tao-whu/e2ec.
1. Introduction
E2EC addresses hand-designed contour initialization, limited feature use, and fixed vertex pairing with learnable initialization, global deformation, multi-direction alignment, and dynamic matching loss.
1. Introduction
- Hand-designed initial shapes can create unreasonable deformation paths, while limited vertex features and fixed label pairings constrain contour adjustment.Fixed pairing may slow convergence or produce wrong predictions.Source: §1 · p. 2
- E2EC replaces manual initialization with a learnable architecture that regresses an initial contour and globally deforms it using contour-vertex features.The architecture includes contour initialization and global contour deformation modules.Source: §1 · p. 2, Figure 2 · p. 2
- Multi-direction alignment fixes selected vertex directions around the center and samples ground-truth vertices between them, restricting pairings and deformation paths.The method aims to reduce learning difficulty while preserving the performance upper bound.Source: §1 · p. 2, Figure 4 · p. 3
- Dynamic matching loss pairs predicted vertices with appropriate label vertices rather than using fixed pairs, improving predicted boundary details.The paper attributes over-smooth boundaries and poor inflection-point fitting to contour-based methods and says DML addresses these problems.Source: §1 · p. 2
- 36 fps on 512×512 images on an NVIDIA A6000 GPU; E2EC reports state-of-the-art performance on KINS, SBD, and Cityscapes.Disabling iterative deformation reaches 50 fps with accuracy comparable to Deep Snake.Source: §1 · p. 2
2. Related Work
Related work contrasts mask-based methods with contour-based approaches, which offer speed but vary in boundary quality and contour construction.
2. Related Work
- Mask-based pipelines can be slow or computationally demanding, while semantic-segmentation-and-clustering methods require complex post-processing and do not support amodal instance segmentation.The section describes both two-stage and one-stage mask-based approaches and a separate semantic-segmentation pipeline.Source: §2 · p. 3
- Contour-based methods are faster than mask-based methods, but center-feature regression can yield rough segmentation, whereas contour-vertex features improve boundary regression.PolarMask and LSNet use center features; Curve GCN, Deep Snake, Point-Set Anchors, and DANCE use contour-vertex features.Source: §2 · p. 3
- Hand-designed contour initialization can produce inappropriate vertex pairs and intersecting deformation paths; E2EC eliminates these paths without reducing the performance upper bound.The section notes that DANCE slightly alleviates path intersections, while Point-Set Anchors' pairing reduces the performance upper bound.Source: §2 · p. 3
- Dense RepPoints requires complex post-processing to obtain a mask or contour, while Polytransform is not end-to-end and is too slow for real-time scenes.Polytransform converts a mask into a contour before refinement.Source: §2 · p. 3
3. The proposed E2EC method
E2EC combines learned contour initialization and deformation with multi-direction alignment and dynamic matching loss. These components shape the contour-learning targets and vertex supervision; the method first locates instance centers, then deforms an initial contour into a final contour.
3. The proposed E2EC method
- E2EC locates instance centers, learns an initial contour, and deforms it through a coarse contour to a final contour.The workflow uses a global deformation module followed by two refinement deformations.Source: §3 · pp. 3–4
3.1. Learnable contour initialization architecture
- The learnable initialization directly regresses an ordered contour from center-point features, while global deformation uses the center and all contour-vertex features to refine it.Unlike Dense RepPoints, the contour is regressed directly without post-processing.Source: §3.1 · p. 4
3.2. Multi-direction alignment (MDA)
- MDA fixes selected vertex directions from the center and uniformly samples ground-truth vertices between them, reducing learning difficulty without lowering the performance upper bound.The experiments identify M = 4 as the best alignment count.Source: §3.2 · p. 4
3.3. Dynamic matching loss (DML)
- DML matches predicted vertices to nearby interpolated boundary points and label key vertices to nearby predictions, improving boundary quality and addressing over-smoothing.Ground-truth segments are split into 10 sub-segments, and key vertices are selected with the Douglas-Peucker algorithm.Source: §3.3 · p. 5
4. Implementation details
The implementation supports multiple detectors and uses CenterNet for comparisons. Smooth L1 supervises contour initialization, global deformation, and the first refinement module, while DML supervises the last refinement module.
4. Implementation details
- E2EC can use any detector by changing the bounding-box branch output size, and the experiments use CenterNet for fair comparison.The output changes from H × W × 2 to H × W × (N × 2) to regress an initial contour with N vertices.Source: §4 · p. 5
- Smooth L1 supervises the contour initialization branch, global deformation branch, and first refinement deformation module.The overall loss adds detection, initialization, coarse-contour, and iterative-refinement losses; α and β are both 0.1.Source: §4 · p. 5
- DML supervises the last deformation module, and the refinement loss combines the losses for its two deformation modules.The passage identifies the first- and second-module outputs as the refinement-stage contour vertices.Source: §4 · p. 5
5. Experiments
Experiments evaluate E2EC on KINS, SBD, Cityscapes, and COCO using mask AP and boundary AP. Ablations test the initialization architecture, alignment sampling, matching loss, and speed–accuracy trade-offs.
5.2. Ablation experiments
- Replacing the baseline initialization with the learnable architecture improves mask and boundary quality; the ablations use SBD train/validation data and a DLA-34 backbone.The paper attributes the gains to a more reasonable initial deformation path.Source: §5.2 · p. 6, Table 1 · p. 6
- Four MDA alignment directions yield the highest APbdy and second-highest APmsk; additional directions reduce learning difficulty but can lower the performance ceiling and worsen vertex distribution.MDA balances learning difficulty against the performance ceiling.Source: §5.2 · p. 6
- DML improves results by 0.4 APmsk and 1.1 APbdy; unlike chamfer loss, it improves boundary quality without damaging vertex order.Chamfer loss changes results by -0.6 APmsk and +0.4 APbdy relative to smooth L1.Source: §5.2 · p. 6
- The coarse contour is predicted 55.5% faster than the final contour, with APmsk reduced by 3.5; full E2EC runs at 36 fps on 512×512 images.The coarse contour matches Deep Snake's final-output accuracy, and the paper presents it as an option when extreme speed is required.Source: §5.2 · pp. 6–7
5.3. Comparison with the state-of-the-art methods
- On KINS, E2EC reaches 36.5 APdet, 34.0 APmsk, and 33.3 APbdy, exceeding Deep Snake by 3.7 APdet, 2.7 APmsk, and 3.1 APbdy.The reported results cover both detection and segmentation.Source: §5.3 · p. 7, Table 6 · p. 7
- On SBD, E2EC segments complex contours such as legs and chairs that Deep Snake does not segment well.The paper notes that E2EC's advantage is more apparent on SBD's more complex contours.Source: §5.3 · pp. 7–8
- With multi-component detection, E2EC scores 39.0 APmsk on Cityscapes validation and 32.9 APmsk on test, exceeding Deep Snake by 1.6 and 1.2 APmsk, respectively.Without multi-component detection, it scores 30.3 APmsk on the test set, 2.1 APmsk above Deep Snake; results are comparable to UPSNet, which used additional COCO training data.Source: Table 5 · p. 7, §5.3 · p. 8
- On COCO test-dev, E2EC achieves 33.8 APmsk at 30.1 fps, while the variant without deformation modules reaches 31.7 APmsk at 54.3 fps.The two variants outperform Deep Snake by 3.5 APmsk and 1.4 APmsk, respectively.Source: Table 3 · p. 6, §5.3 · p. 8
6. Conclusion
E2EC combines learnable contour initialization, multi-direction alignment, and dynamic matching loss for end-to-end contour-based instance segmentation. The paper reports state-of-the-art results on KINS, SBD, and Cityscapes, beyond real-time performance, and a faster variant.
6. Conclusion
- E2EC introduces learnable contour initialization, multi-direction alignment, and dynamic matching loss for contour-based instance segmentation.The paper reports improved contour extraction quality.Source: §6 · p. 8
- E2EC achieves state-of-the-art results on KINS, SBD, and Cityscapes with beyond real-time performance.Source: §6 · p. 8
- A faster variant retains only the learnable contour initialization architecture, with accuracy comparable to Deep Snake and speed almost matching CenterNet.The paper says the proposed modules could be applied to other contour-based instance segmentation methods.Source: §6 · p. 8