Source-linked AI summary
SCNet: Training Inference Sample Consistency for Instance Segmentation
Thang Vu, Haeyong Kang, Chang D. Yoo
TL;DR
Cascade architectures improve detection and instance segmentation, but training and inference can use different IoU distributions for mask samples. SCNet aligns those distributions and adds feature relay and global context; on COCO it improves box and mask AP over Cascade Mask R-CNN while running faster.
Problem
Cascade Mask R-CNN trains mask prediction on outputs from all box stages but uses only the last stage at inference, creating a sample-distribution mismatch that can potentially worsen performance.
Source: §1 · p. 1Method
SCNet uses only the last box stage for mask predictions during both training and inference, and adds feature relay and global context to strengthen relationships among sub-tasks.
Source: §1 · p. 1Results
SCNet runs faster and improves box AP by 1.3 points and mask AP by 2.3 points versus the Cascade Mask R-CNN baseline on COCO.
Source: §6 · p. 7Takeaways & Limitations
On COCO, SCNet combines closer training–inference sample distributions with stronger sub-task relationships and better box and mask AP than the Cascade Mask R-CNN baseline.
Source: §1 · p. 1, §6 · p. 7HTC retains sample inconsistency and resource-intensive multi-stage mask prediction and mask ensemble.
Source: §3.2 · p. 4
Abstract
from arXiv · showhide
Cascaded architectures have brought significant performance improvement in object detection and instance segmentation. However, there are lingering issues regarding the disparity in the Intersection-over-Union (IoU) distribution of the samples between training and inference. This disparity can potentially exacerbate detection accuracy. This paper proposes an architecture referred to as Sample Consistency Network (SCNet) to ensure that the IoU distribution of the samples at training time is close to that at inference time. Furthermore, SCNet incorporates feature relay and utilizes global contextual information to further reinforce the reciprocal relationships among classifying, detecting, and segmenting sub-tasks. Extensive experiments on the standard COCO dataset reveal the effectiveness of the proposed method over multiple evaluation metrics, including box AP, mask AP, and inference speed. In particular, while running 38\% faster, the proposed SCNet improves the AP of the box and mask predictions by respectively 1.3 and 2.3 points compared to the strong Cascade Mask R-CNN baseline. Code is available at \url{https://github.com/thangvubk/SCNet}.
1 Introduction
Cascade Mask R-CNN improves instance segmentation, but its training and inference use different mask-sample distributions. SCNet aligns those distributions and adds feature relay and global context to connect segmentation with detection and classification.
1 Introduction
- Training uses outputs from all box stages for mask prediction, whereas inference uses only the last stage, creating a sample-distribution mismatch.The differing IoU thresholds across box stages produce different sample distributions, potentially worsening performance.Source: §1 · p. 1
- 49.1% of training samples without consistency have IoU ≤ 0.75, versus 34.2% with consistency and 38.5% at inference.Training with sample consistency therefore produces a closer IoU distribution to inference than training without it.Source: §1 · p. 1
- Feature relay explicitly connects box-branch output to mask-branch input, while global context supplies object context for final predictions.These components reinforce relationships among detecting, classifying, and segmenting objects; region-wise pooling otherwise focuses prediction on small image regions.Source: §1 · pp. 1–2
2 Related Work
Instance-segmentation approaches include proposal-based and proposal-free methods, with proposal-based methods generally more accurate. SCNet addresses limitations of cascaded approaches through sample consistency, feature relay, and instance-level global context.
Instance Segmentation
- Instance segmentation includes proposal-based methods that generate regions before predicting masks and proposal-free methods that identify instances without proposals.The related-work overview also describes Cascade Mask R-CNN and HTC as extensions of multi-stage detection for instance segmentation.Source: Instance Segmentation · p. 2
Related approaches
- Proposal-based methods generally achieve greater accuracy, while proposal-free methods are described as simple and fast.Proposal-based methods predict masks within generated regions; proposal-free methods identify instances directly without proposals.Source: Instance Segmentation · p. 2
Multi-stage Instance Segmentation
- Cascade Mask R-CNN and HTC improve on non-cascade methods but retain training–inference sample inconsistency and computationally costly multi-stage mask prediction.SCNet instead aligns training and testing samples and avoids repeated mask RoI feature extraction, feature upsampling, and mask prediction.Source: Multi-stage Instance Segmentation · pp. 2–3
- Feature relay creates information flow between box and mask branches, while global context supplies each object with a context prior for final prediction.The proposed global context operates at instance level in detector stages, unlike cited prior methods that incorporate context in the backbone at pixel level.Source: Multi-stage Instance Segmentation · p. 3
3 Cascade Architectures
Cascade Mask R-CNN combines Cascade R-CNN detection with Mask R-CNN segmentation, while HTC interleaves box and mask processing. Both retain limitations that motivate sample-consistent mask prediction and more efficient connections between branches.
3.1 Cascade Mask R-CNN
- Cascade Mask R-CNN combines a high-performing Cascade R-CNN detector with Mask R-CNN segmentation.Its mask branches use all box-stage outputs during training but only the final box-stage output during inference, producing inconsistent sample distributions.Source: §3.1 · pp. 3–4
- Cascade Mask R-CNN also has isolated mask branches, which make predictions on noisy intermediate boxes.Its performance relies chiefly on the detector and an ensemble of multiple isolated mask branches.Source: §3.1 · p. 4
3.2 Hybrid Task Cascade
- HTC interleaves box and mask execution, using the regressed box at stage t for segmentation and passing accumulated mask features between stages.It nevertheless retains sample inconsistency and resource-intensive multi-stage mask prediction and mask ensemble.Source: §3.2 · p. 4
4 The proposed SCNet
SCNet aligns training and inference mask samples with an efficient single deep mask branch, then couples detection and segmentation through feature relay and global context. It is trained end-to-end with a multi-task loss that includes global-context supervision.
4.1 Sample Consistency
- Effective sample consistency uses the last box stage and one deep mask branch, stacking three four-convolution branches into 12 layers with skip connections every two layers.This avoids repeated expensive operations and intermediate noisy-box mask predictions; naive consistency instead retains multiple shallow branches and repeated operations.Source: §4.1 · p. 4
4.2 Feature Relay and Global Context
- Feature relay transforms positive-sample box features and fuses them with same-stage mask features, while mask supervision refines box features through back-propagation.The box features are aligned, reshaped, and upsampled before element-wise summation with mask features; with sample consistency, relay is applied only at the last stage.Source: §4.2 · p. 4, Figure 4 · p. 5
- The global context branch processes top-level FPN features to produce multi-label predictions and context features for fusion with box and mask features.The branch uses two residual blocks followed by global average pooling; its multi-label predictions are supervised by known objects in the image.Source: §4.2 · pp. 4–5, Figure 5 · p. 5
4.3 Training
- SCNet is trained end-to-end with a multi-task loss, including classification, regression, mask, semantic, and global-context losses.The single mask branch's loss weight is reweighted to equal the sum of stage-wise mask-loss weights, and global-context classification uses binary cross entropy.Source: §4.3 · p. 5
5 Experiments
On COCO, SCNet outperforms cascade baselines in box AP, mask AP, and inference speed. Ablations report gains from sample consistency, feature relay, and global context, alongside the experimental setup.
5.1 Implementation Details
- Experiments resize images to 1333 by 800 pixels, train for 20 epochs on eight GPUs, and evaluate AP using COCO style; runtime is measured on one Tesla V100 GPU.Training uses standard horizontal flips and takes about one day on eight Tesla V100 GPUs.Source: §5.1 · p. 5
5.2 Benchmarking Results
- With ResNet-50, SCNet improves box AP by 1.3 points and mask AP by 2.3 points over Cascade Mask R-CNN, while running 1.7 fps (38%) faster than Cascade Mask R-CNN and HTC.SCNet achieves 6.2 fps; Table 1 benchmarks methods on COCO test-dev, with AP denoting mask AP and APbb denoting box AP.Source: §5.2 · p. 5, Table 1 · p. 6
5.3 Ablation Study
- Combining all components yields mask AP 39.8 and box AP 33.6, with improvements of 2.4 mask AP points, 1.3 box AP points, and 1.7 fps over the baseline.The baseline is Cascade Mask R-CNN with the semantic branch.Source: Component-wise Analysis · p. 6, Table 2 · p. 6
- Effective sample consistency improves mask AP while speeding inference by avoiding repeated expensive operations through stages.Naive sample consistency raises mask AP from 37.4 to 38.0 but keeps speed unchanged at 4.5 fps because it still requires multiple mask branches.Source: Component-wise Analysis · pp. 6–7, Table 3 · p. 6
- Feature relay raises mask AP by 0.6 points in both Mask R-CNN and Cascade Mask R-CNN, with marginal computational overhead and comparable box AP.The relay fuses adapted box features with mask features and targets mask prediction.Source: Feature Relay · p. 7, Table 4 · p. 7
- Setting the global context loss weight to 3 achieves the best overall performance, yielding a 0.8-point mask AP improvement; performance increases for positive weights and is not sensitive to the loss weight.Source: Global Context · p. 7
6 Conclusion
SCNet aligns training and inference sample distributions while speeding the network, and reinforces relationships among instance-segmentation subtasks. On COCO, it improves box and mask AP over Cascade Mask R-CNN while running faster.
6 Conclusion
- On COCO, SCNet improves box AP by 1.3 points and mask AP by 2.3 points over Cascade Mask R-CNN while running faster.The paper reports these improvements against the strong Cascade Mask R-CNN baseline.Source: §6 · p. 7
- SCNet ensures that the IoU distribution of samples at training time is close to that at inference time while speeding up the network.Source: §6 · p. 7
- Feature relay and global context strengthen relationships among classifying, detecting, and segmenting subtasks.Source: §6 · p. 7