Source-linked AI summary
CascadePSP: Toward Class-Agnostic and Very High-Resolution Segmentation via Global and Local Refinement
Ho Kei Cheng, Jihoon Chung, Yu-Wing Tai, Chi-Keung Tang
TL;DR
Low-resolution segmentation models struggle with very high-resolution images because upsampling misses boundary details, while high-resolution training data are unavailable or impractical. CascadePSP refines arbitrary input masks through global and local coarse-to-fine processing, using only low-resolution training data. The method produces high-resolution, pixel-accurate boundary refinements without finetuning and extends to multi-class scene parsing, though large input errors can remain unresolved.
Problem
Very high-resolution segmentation remains challenging because low-resolution outputs miss boundary details and high-resolution training data are difficult to obtain and process.
Method
CascadePSP refines any input segmentation through a coarse-to-fine cascade combining global structural refinement with local full-resolution boundary refinement.
Results
CascadePSP performs segmentation refinement up to 4K using a single low-resolution-trained model without finetuning, with consistent improvements across datasets and models.
Takeaways & Limitations
The framework provides class-agnostic high-resolution refinement and supports high-quality multi-class scene parsing without high-resolution training images.
Takeaways & Limitations
The method relies on the input segmentation and low-level cues, so sufficiently large input errors may remain unresolved.
Abstract
from arXiv · showhide
State-of-the-art semantic segmentation methods were almost exclusively trained on images within a fixed resolution range. These segmentations are inaccurate for very high-resolution images since using bicubic upsampling of low-resolution segmentation does not adequately capture high-resolution details along object boundaries. In this paper, we propose a novel approach to address the high-resolution segmentation problem without using any high-resolution training data. The key insight is our CascadePSP network which refines and corrects local boundaries whenever possible. Although our network is trained with low-resolution segmentation data, our method is applicable to any resolution even for very high-resolution images larger than 4K. We present quantitative and qualitative studies on different datasets to show that CascadePSP can reveal pixel-accurate segmentation boundaries using our novel refinement module without any finetuning. Thus, our method can be regarded as class-agnostic. Finally, we demonstrate the application of our model to scene parsing in multi-class segmentation.
1. Introduction
High-resolution semantic segmentation is difficult because low-resolution models lose boundary detail, while high-resolution training data and computation are impractical. CascadePSP addresses this by refining arbitrary input masks through a coarse-to-fine cascade and reports consistent improvements without dataset-specific finetuning.
- Motivation: 4K UHD segmentation is difficult because memory scales linearly with pixel count and pixel-accurate high-resolution training annotations are scarce.Directly training on 4K UHD images is described as practically impossible under these constraints.
- Motivation: Downsampling removes image details, whereas cropping destroys image context, leaving neither workaround adequate for very high-resolution segmentation.
- CascadePSP: CascadePSP refines any input segmentation from low to high resolution using a coarse-to-fine cascade whose early structural predictions guide later boundary refinement.
- CascadePSP: The method corrects erroneous boundaries and produces very-high-resolution outputs without requiring high-resolution training images or finetuning.
- Evaluation: Across PASCAL VOC 2012, BIG, and ADE20K, one model achieved consistent improvement over state-of-the-art methods without finetuning on the evaluated dataset.The BIG dataset contains 50 validation and 100 test objects with PASCAL-matched semantic classes.
2. Related Works
Prior segmentation systems use contextual architectures and often recover sharper boundaries through encoder-decoder designs, but output-stride upsampling remains inaccurate at boundaries. Existing global-local refinement addresses this issue but requires high-resolution training images, whereas CascadePSP combines encoder-decoder refinement with a cascade for high-resolution generalization.
- Context modeling: Segmentation methods capture context through wide field-of-view, image-pyramid, or feature-pyramid architectures to support pixel labeling.
- Encoder-decoder methods: Encoder-decoder models reduce spatial dimensions for semantics, then restore spatial extent; skip connections can sharpen recovered boundaries.
- Resolution limits: Large output strides reduce memory and computation, but bilinear upsampling to target size produces inaccurate boundary labels.
- Prior refinement: Global-Local Networks use global information and local fine-structure branches, but still require high-resolution training images unavailable for many tasks.
- CascadePSP: CascadePSP adopts encoder-decoder refinement with a cascade to generate high-resolution segmentations and generalize without finetuning.
3. CascadePSP
CascadePSP refines segmentation progressively with multi-scale inputs and coarse-to-fine cascades. Its Global and Local steps combine whole-image structure with high-resolution crop details for boundary refinement.
- Refinement Module: The refinement module combines an image with imperfect masks at multiple scales to produce progressively finer segmentation outputs.Multi-scale inputs capture structural and boundary information, which the network adaptively fuses at the finest level.
- Refinement Module: The module generates stride 8, stride 4, and stride 1 outputs, using coarse predictions for structure and fine predictions for boundaries.Stride 2 is skipped to preserve flexibility for correcting local boundary errors.
- Refinement Module: A 3-level cascade captures structural cues from small-scale intermediates, enabling more detailed segmentation than a 1-level model with the same receptive field.The intermediate segmentations may be inaccurate but can preserve object structure for later refinement.
- Global and Local Refinement: The Global step refines a resized whole image, while the Local step processes high-resolution crops and fuses their outputs for final refinement.The same trained refinement module can be reused recursively for higher-resolution inputs.
- Global and Local Refinement: Both Global and Local refinement are essential: the Global step mainly repairs overall structure, whereas the Local step improves high-resolution boundary accuracy.The Local step is crucial above the 900-pixel switching point, while lower-resolution inputs use only the Global step.
4. Experiments
Experiments evaluate CascadePSP without finetuning on relabeled PASCAL VOC, the high-resolution BIG dataset, and ADE20K, including object refinement and scene parsing. Results show improved segmentation quality and high-resolution refinement, while errors lacking semantic information remain a limitation.
- 4. Experiments: The evaluation uses PASCAL VOC 2012, BIG, and ADE20K, with experiments conducted without finetuning.The study also evaluates refinement across different input models and scene-parsing settings.
- 4.1. Dataset and Evaluation Method: 500 PASCAL VOC validation segmentations were relabeled to provide accurate boundaries within regions previously marked “void”.The relabeled set supports more accurate boundary evaluation.
- 4.1. Dataset and Evaluation Method: BIG contains 50 validation and 100 test objects at resolutions from 2048×1600 to 5000×3600, labeled without void regions.Evaluation uses IoU and the boundary metric mBA.
- 4.2. PASCAL VOC 2012: CascadePSP improves segmentation quality in all tested PASCAL VOC cases, especially along object boundaries.The method is compared with several semantic segmentation outputs and is reported as more effective than commonly used multiscale testing.
- 4.3. BIG dataset: On BIG, CascadePSP produces high-quality refinements at high resolution despite never seeing high-resolution training images.Unlike super-resolution upsampling, refinement can address some erroneous input segmentations, although it cannot recover missing semantic information in every case.
- 4.4. Scene parsing: Scene parsing uses divide-and-conquer refinement of semantic objects followed by fusion, with padded ROIs and a modified argmax for overlapping regions.Sufficiently large connected components are refined independently using ROIs with 25% padding.
5. Conclusion
The conclusion presents CascadePSP as a general refinement framework that improves input segmentations without finetuning and supports high-resolution refinement from low-resolution training data. Its Global step supplies image context for Local full-resolution refinement, including up to 4K inputs.
- 5. Conclusion: CascadePSP refines arbitrary input segmentations without finetuning and performs high-resolution refinement up to 4K.The model is trained on low-resolution data and has never seen high-resolution training images.