Source-linked AI summary
Hybrid Task Cascade for Instance Segmentation
Kai Chen, Jiangmiao Pang, Jiaqi Wang, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jianping Shi, Wanli Ouyang, Chen Change Loy, Dahua Lin
TL;DR
Instance segmentation cascades remain difficult because naive combinations yield limited mask gains and insufficient information flow. HTC interweaves detection and segmentation in joint multi-stage processing while adding a fully convolutional spatial-context branch. It improves mask AP over strong baselines and reaches 48.6 mask AP on COCO test-challenge and 49.0 on test-dev.
Problem
Naively combining Cascade R-CNN and Mask R-CNN provides limited mask gains, with later mask stages lacking direct connections for information flow.
Method
HTC interweaves box and mask branches across stages, passes mask features forward, and jointly uses a fully convolutional semantic-segmentation branch for spatial context.
Results
48.6 mask AP on COCO test-challenge and 49.0 mask AP on test-dev; HTC also improves 1.5% over a strong Cascade Mask R-CNN baseline on MSCOCO.
Takeaways & Limitations
Joint multi-stage task processing and complementary spatial-context features support progressively refined, more accurate instance-segmentation predictions.
Takeaways & Limitations
Some module implementations and integration methods may not be optimal and warrant further study.
Abstract
from arXiv · showhide
Cascade is a classic yet powerful architecture that has boosted performance on various tasks. However, how to introduce cascade to instance segmentation remains an open question. A simple combination of Cascade R-CNN and Mask R-CNN only brings limited gain. In exploring a more effective approach, we find that the key to a successful instance segmentation cascade is to fully leverage the reciprocal relationship between detection and segmentation. In this work, we propose a new framework, Hybrid Task Cascade (HTC), which differs in two important aspects: (1) instead of performing cascaded refinement on these two tasks separately, it interweaves them for a joint multi-stage processing; (2) it adopts a fully convolutional branch to provide spatial context, which can help distinguishing hard foreground from cluttered background. Overall, this framework can learn more discriminative features progressively while integrating complementary features together in each stage. Without bells and whistles, a single HTC obtains 38.4 and 1.5 improvement over a strong Cascade Mask R-CNN baseline on MSCOCO dataset. Moreover, our overall system achieves 48.6 mask AP on the test-challenge split, ranking 1st in the COCO 2018 Challenge Object Detection Task. Code is available at: https://github.com/open-mmlab/mmdetection.
1. Introduction
Instance segmentation must handle appearance variation and cluttered backgrounds, yet naive cascaded designs provide limited mask gains. HTC addresses this by interweaving detection and segmentation, adding spatial context, and progressively integrating features.
- Motivation: Instance segmentation requires robust representations for deformation, occlusion, scale changes, and cluttered backgrounds.Accurate per-pixel instance labeling is difficult in real-world scenarios because objects vary in appearance and may be hard to isolate.
- Motivation: Cascade R-CNN and Mask R-CNN combined naively improve bbox AP by 3.5% but mask AP by only 1.2%.Later mask stages benefit from better localized boxes but lack direct connections between mask branches.
- HTC: HTC interweaves bounding-box regression and mask prediction in a joint multi-stage pipeline with direct mask-feature connections across stages.This design strengthens information flow between tasks and across stages for progressive refinement.
- HTC: A fully convolutional branch supplies spatial context from foreground and background regions to complement box and mask features.The semantic context branch is intended to help distinguish hard foreground from cluttered background.
- Results: HTC achieves 2.6% and 1.4% higher mask AP than Mask R-CNN and Cascade Mask R-CNN baselines, respectively, on COCO.With stronger backbones and common components, the overall system reaches 49.0 mask AP on COCO test-dev.
- Contribution: The framework integrates cascade and multi-tasking to interweave detection and segmentation features for joint multi-stage processing.The study also reports state-of-the-art performance on COCO test-dev and test-challenge.
2. Related Work
Prior instance-segmentation methods are broadly detection-based or segmentation-based, while multi-stage detection provides a cascade precedent. HTC combines detection, mask prediction, semantic segmentation, contextual information, and cross-stage flow in one pipeline.
- Instance Segmentation: Instance segmentation localizes objects at pixel level using masks and is closely related to object detection and semantic segmentation.Existing approaches therefore broadly divide into detection-based and segmentation-based methods.
- Instance Segmentation: Detection-based methods generate boxes or proposals before predicting masks within them, often using CNN-based pipelines trained end-to-end.Examples include DeepMask, SharpMask, InstanceFCN, and MNC.
- Instance Segmentation: Segmentation-based methods first produce image-level pixel segmentation and then identify object instances from that result.Representative approaches use local patches, MRFs, CRFs, or watershed-based energy maps.
- Multi-stage Object Detection: Multi-stage object detection emerged as a major paradigm through iterative localization and refinement mechanisms.Cascade architectures build on progressive refinement across stages.
- HTC: HTC combines detection, mask prediction, and semantic segmentation at each stage to exploit reciprocal task relationships.It additionally uses a stuff-segmentation branch for context and a direct path for information flow across stages.
3. Hybrid Task Cascade
Hybrid Task Cascade integrates detection and segmentation through joint multi-stage processing, direct mask-feature connections, and a semantic segmentation branch that adds spatial context.
- Multi-task Cascade: HTC interleaves bounding-box regression and mask prediction instead of executing the two branches in parallel.This lets mask prediction use updated bounding-box predictions within each stage.
- Mask Information Flow: Direct connections feed intermediate mask features from each stage into the next, enabling progressive mask refinement.The connected mask branches are jointly supervised through backpropagation.
- Spatial Contexts from Segmentation: A fully convolutional semantic branch predicts whole-image pixel-level segmentation and supplies contextual features to box and mask heads.The branch uses semantic information from foreground and background regions, helping distinguish objects from cluttered backgrounds.
- Semantic Segmentation Branch: The semantic branch combines feature-pyramid levels to incorporate higher-level global information and lower-level local information.Features are aligned, resized to a common scale, fused by element-wise summation, and used to predict a pixel-wise segmentation map.
- Learning: HTC is trained end-to-end with a multi-task loss combining box, mask, and semantic-segmentation objectives across stages.The box loss combines classification and regression terms, while mask and semantic losses use binary cross entropy and cross entropy, respectively.
4. Experiments
Experiments evaluate HTC on COCO using standardized training and inference settings, benchmark comparisons, and component ablations. HTC improves consistently over Cascade Mask R-CNN, while interleaving, mask information flow, semantic context, and feature fusion each contribute measurable gains.
- Experimental Setup: Experiments use COCO 2017 train, val, and test-dev splits with box AP and mask AP evaluated under standard COCO-style metrics.Models use a 3-stage cascade and FPN, with fixed training and image-resizing protocols for fair comparison.
- Benchmarking Results: HTC improves mask AP by 1.5%, 1.3%, and 1.1% over the cascade baseline with ResNet-50, ResNet-101, and ResNeXt-101 backbones, respectively.The gains are reported across different backbones on COCO.
- Component-wise Analysis: 0.2% mask AP comes from interleaved execution, followed by 0.6% from mask information flow and 0.6% from the semantic segmentation branch.These effects are measured in the component-wise analysis.
- Interleaved Branch Execution: Interleaved execution improves mask AP by 0.5% for Mask R-CNN and 0.2% for Cascade Mask R-CNN compared with parallel execution.The comparison tests both methods under the alternative branch-execution strategies.
- Semantic Feature Fusion: 0.4% additional gain results from fusing semantic features with both box and mask branches, while performance is insensitive to the semantic-branch loss weight.The fusion result supports combining complementary features across branches.
- Extensions on HTC: 49.0 mask AP is achieved on COCO test-dev with stronger backbones and additional components including deformable convolution, multi-scale processing, and ensembling.This overall system is 2.3% higher than the winning COCO Challenge 2017 approach.
5. Conclusion
HTC interweaves box and mask branches in joint multi-stage processing and adds semantic context. It improves over Cascade Mask R-CNN and reaches strong COCO test-dev and test-challenge mask AP results.
- Conclusion: HTC interweaves box and mask branches for joint multi-stage processing and uses a semantic segmentation branch to provide spatial context.The framework progressively refines masks while integrating complementary features at each stage.
- Conclusion: 1.5% improvement over a strong Cascade Mask R-CNN baseline is obtained without bells and whistles on MSCOCO.The comparison concerns mask AP.
- Conclusion: 48.6 mask AP is achieved on the COCO test-challenge dataset, while 49.0 mask AP is achieved on test-dev.These are reported results for the overall system.