Source-linked AI summary
SegFix: Model-Agnostic Boundary Refinement for Segmentation
Yuhui Yuan, Jingyi Xie, Xilin Chen, Jingdong Wang
TL;DR
Segmentation models often make unreliable predictions near object boundaries. SegFix learns image-based boundary and direction maps to replace boundary labels with corresponding interior labels, and it consistently improves diverse models across multiple benchmarks while requiring no model-specific prior information.
Problem
Existing segmentation models often produce unreliable predictions along object boundaries, although interior-pixel predictions are more reliable.
Method
SegFix predicts boundary pixels and image-derived directions toward corresponding interior pixels, then uses the resulting offsets to refine any existing segmentation result.
Results
SegFix consistently improves various segmentation models across Cityscapes, ADE20K, and GTA5, including a 1.7% boundary F-score improvement when applied to Gated-SCNN.
Takeaways & Limitations
SegFix generalizes across strong semantic and instance segmentation models and can serve as a baseline for more accurate boundary results.
Takeaways & Limitations
Boundary pixels are defined by neighboring pixels with different categories, with ground-truth boundary width generated using a distance-transform threshold.
Abstract
from arXiv · showhide
We present a model-agnostic post-processing scheme to improve the boundary quality for the segmentation result that is generated by any existing segmentation model. Motivated by the empirical observation that the label predictions of interior pixels are more reliable, we propose to replace the originally unreliable predictions of boundary pixels by the predictions of interior pixels. Our approach processes only the input image through two steps: (i) localize the boundary pixels and (ii) identify the corresponding interior pixel for each boundary pixel. We build the correspondence by learning a direction away from the boundary pixel to an interior pixel. Our method requires no prior information of the segmentation models and achieves nearly real-time speed. We empirically verify that our SegFix consistently reduces the boundary errors for segmentation results generated from various state-of-the-art models on Cityscapes, ADE20K and GTA5. Code is available at: https://github.com/openseg-group/openseg.pytorch.
1 Introduction
SegFix targets thin-boundary segmentation errors by replacing unreliable boundary labels with labels from corresponding interior pixels. It learns boundary localization and image-based interior correspondences without using segmentation outputs.
- Contribution: SegFix improves segmentation results on pixels located within the thinning boundary through model-agnostic boundary refinement.The method is designed to operate after an existing segmentation model produces its result.
- Motivation: Segmentation errors frequently concentrate along thin object boundaries, while interior-pixel predictions are more reliable.Across DeepLabv3, HRNet, and Gated-SCNN, error counts decrease as distance from boundaries increases.
- Method: The method replaces boundary-pixel labels with labels from corresponding interior pixels identified using only the input image.It first predicts a binary boundary mask, then learns a direction from each boundary pixel toward an interior pixel.
- Evaluation: SegFix consistently improves various segmentation models across Cityscapes, ADE20K, and GTA5 without prior information about those models.The method is also extended to instance segmentation on Cityscapes.
- Evaluation: 84.5% and 41.2% are reported for HRNet + OCR + SegFix and PolyTransform + SegFix on the semantic and instance segmentation tracks, respectively.These results ranked first and second according to the Cityscapes leaderboard by the ECCV 2020 submission deadline.
- Boundary definition: The boundary definition treats pixels with neighboring pixels from different categories as boundary pixels.Ground-truth boundary maps are generated with a distance transform and a chosen width.
2 Related Work
SegFix differs from segmentation-integrated boundary methods by refining existing segmentation maps directly from image-derived offsets. It is designed to avoid model-specific retraining while remaining complementary to prior approaches.
- Distance/Direction Map for Segmentation: Prior distance- or direction-map methods commonly predict auxiliary maps within segmentation or instance-segmentation frameworks.SegFix instead predicts a direction map from only the image and uses it to refine existing segmentation results.
- Boundary-aware Segmentation: Several boundary-aware methods depend on the segmentation model and require careful retraining or fine-tuning.SegFix does not perform segmentation prediction or feature propagation, and directly refines maps with an offset map.
- Boundary-aware Segmentation: A single unified SegFix model can be trained once and applied across different segmentation models without further fine-tuning.The paper reports that SegFix is complementary to methods including Gated-SCNN and Boundary-Aware Feature Propagation.
- Guided Up-sampling Network: Guided up-sampling methods require retraining for different models and do not explicitly supervise their offset maps.SegFix applies explicit semantic-aware supervision and transfers its offset maps directly across approaches.
- STEAL: STEAL uses mean-squared loss for its direction branch, whereas SegFix uses cross-entropy loss on discrete directions.The paper compares the two methods in an ablation study.
3 Approach
SegFix trains an image-only model to locate boundary pixels and predict offsets toward interior pixels, then uses those offsets to refine coarse segmentation maps from any existing model. Its pipeline derives boundary and direction supervision from distance transforms and converts discrete directions into offsets for label shifting.
- Framework: SegFix predicts boundary and direction maps from the input image alone, without directly performing segmentation during training.The predicted offsets identify interior pixels used to refine outputs from existing segmentation models.
- Framework: A boundary pixel at coordinate p_i is relabeled from the coarse map using the pixel at p_i + ∆q_i, where ∆q_i points toward an interior pixel.The method applies the generated offset map to the coarse label map; rescaling offsets is the default solution for thick boundaries.
- Framework: The same test-time offset maps can refine outputs from different segmentation models without prior model information or further fine-tuning.This makes SegFix model-agnostic across existing segmentation models and datasets.
- Network Architecture: The direction map is masked by the binarized boundary map so direction loss is applied only at pixels identified as boundaries.Boundary and direction branches use separate binary and categorical cross-entropy losses.
- Network Architecture: Direction categories are mapped to discrete offsets; for m = 4, the four angular ranges correspond to (1, 1), (−1, 1), (−1, −1), and (1, −1).The offset branch converts the predicted direction map into a two-channel offset map and shifts labels with grid sampling.
- Ground-Truth Generation: Ground-truth maps are constructed by decomposing labels into category-specific binary maps, applying distance transforms, fusing distances, and deriving boundary and direction maps.Pixels with fused distance below the threshold γ are marked as boundary; the implementation uses a small value such as γ = 5 for thin-boundary refinement.
4 Experiments: Semantic Segmentation
SegFix is evaluated as a model-agnostic refinement scheme across semantic-segmentation benchmarks, backbones, boundary settings, and competing methods. Experiments report consistent gains in overall and boundary quality, with unified training and near-real-time operation.
- Implementation and operation: SegFix requires no extra training or fine-tuning of semantic-segmentation models; it predicts boundary masks and direction maps from test images before refinement.The predicted offsets identify corresponding pixels used to refine results from existing approaches.
- Datasets and metrics: SegFix is evaluated on Cityscapes, ADE20K, and GTA5, using mIoU for overall segmentation and boundary F-score for boundary quality.Cityscapes includes semantic and instance-segmentation settings; ADE20K contains 150 categories, while GTA5 has 19 categories compatible with Cityscapes.
- Implementation and operation: Real-time speed is achieved with lightweight backbones such as HRNet-W18 and HRNet-W32 on 2048×1024 inputs.Runtime is measured on a single V100 GPU using PyTorch1.4 and TensorRT.
- Ablation experiments: SegFix consistently improves both segmentation performance and boundary quality across HRNet-W18, HRNet-W32, and HRNet-2× backbones.HRNet-2× is selected for subsequent experiments because it performs best in the backbone ablation.
- Comparisons and applications: 1.7% boundary F-score improvement is obtained when SegFix refines Gated-SCNN results, while unified training yields 0.9%/3.8% gains on Cityscapes and 0.5%/2.7% on ADE20K by mIoU/F-score.SegFix also outperforms GUM and DenseCRF, remains comparable to STEAL for boundary detection, and has computation cost similar to a two-model ensemble.
- Cross-benchmark results: 2.9%/11.5% gains in boundary F-score are reported for DeepLabv3 on ADE20K val/GTA5 test, respectively.The boundary F-score improvements are measured on the two benchmarks using the paper’s boundary-quality evaluation.
5 Experiments: Instance Segmentation
SegFix consistently improves Cityscapes instance-segmentation performance across several existing models, with larger gains for object-focused evaluation than semantic segmentation.
- SegFix consistently improves mean AP for Mask-RCNN, PANet, PointRend, and PolyTransform on Cityscapes instance segmentation.
- 1.4 percentage points: PANet gains this amount on the Cityscapes test set with SegFix.
- 1.5% and 1.1%: SegFix improves PointRend and PolyTransform performance by these amounts, respectively.
- Instance-segmentation improvements of +1.1% to +1.5% exceed semantic-segmentation improvements of +0.3% to +0.5%.
- The authors attribute the larger instance-segmentation gains partly to evaluation covering eight object categories while excluding stuff categories.
6 Conclusion
SegFix refines segmentation maps from unknown models by replacing boundary predictions with corresponding interior predictions learned from the input image, and generalizes across tasks and models.
- SegFix refines maps from an unknown segmentation model by replacing boundary predictions with corresponding interior predictions.
- The method learns boundary-to-interior correspondences using only the input image.
- Empirical results support SegFix's effectiveness for both semantic and instance segmentation tasks.
- The authors propose SegFix as a strong baseline for more accurate segmentation results along boundaries.
7 Appendix
The appendix details category-level behavior, unified training, instance-segmentation application, and qualitative improvements, while noting that interior errors remain outside SegFix's main focus.
- 7.1 Boundary-pixel statistics: Boundary pixels occupy large proportions in the small-scale pole, traffic-light, and traffic-sign categories.
- 7.2 Category-wise mIoU improvements: 3.1%, 2.7%, and 2.4%: SegFix improves DeepLabv3 mIoU for pole, traffic light, and traffic sign, respectively.
- 7.2 Category-wise mIoU improvements: SegFix significantly improves object categories including pole, traffic light, and traffic sign across DeepLabv3, Gated-SCNN, and HRNet.
- 7.3 Details of Unified SegFix Experiments: Unified SegFix uses mixed Cityscapes and ADE20K mini-batches and achieves performance comparable to training separately on each dataset.
- 7.4 Cityscapes Instance Segmentation: For instance segmentation, SegFix generates offset maps from instance masks and applies them to refine predictions from existing models.
- 7.5 More Qualitative Results: Qualitative results show that SegFix addresses thin-boundary errors but leaves some errors in interior regions.