Source-linked AI summary
Progressive Semantic Segmentation
Chuong Huynh, Anh Tran, Khoa Luu, Minh Hoai
TL;DR
High-resolution segmentation must work within GPU memory limits without losing fine details or the global context needed to resolve ambiguous patches. MagNet uses multi-scale stages to progressively refine segmentation from coarse to fine. Across three challenging high-resolution datasets, it outperformed previous state-of-the-art methods by 1% to 2% in mIoU.
Problem
High-resolution segmentation is constrained by GPU memory, while downsampling loses detail and independent patch processing lacks global information for resolving local ambiguity.
Source: §1 · p. 1Method
Results
MagNet outperformed previous state-of-the-art methods by 1% to 2% in mean Intersection over Union (mIoU) on three challenging high-resolution image datasets.
Source: §5 · p. 8Takeaways & Limitations
MagNet generates high-resolution segmentation output without exploding GPU memory usage by dividing input images into patches and processing multiple scales.
Source: §5 · p. 8The segmentation backbone must output a segmentation map with uncertainty estimates.
Source: §3 · p. 3
Abstract
from arXiv · showhide
The objective of this work is to segment high-resolution images without overloading GPU memory usage or losing the fine details in the output segmentation map. The memory constraint means that we must either downsample the big image or divide the image into local patches for separate processing. However, the former approach would lose the fine details, while the latter can be ambiguous due to the lack of a global picture. In this work, we present MagNet, a multi-scale framework that resolves local ambiguity by looking at the image at multiple magnification levels. MagNet has multiple processing stages, where each stage corresponds to a magnification level, and the output of one stage is fed into the next stage for coarse-to-fine information propagation. Each stage analyzes the image at a higher resolution than the previous stage, recovering the previously lost details due to the lossy downsampling step, and the segmentation output is progressively refined through the processing stages. Experiments on three high-resolution datasets of urban views, aerial scenes, and medical images show that MagNet consistently outperforms the state-of-the-art methods by a significant margin.
1. Introduction
High-resolution segmentation is constrained by GPU memory, while downsampling and independent patch processing each sacrifice information needed for accurate, detailed maps. MagNet addresses this trade-off with multi-scale progressive refinement that combines global context with higher-resolution detail.
1. Introduction
- High-resolution input images often exceed the memory and computational capacity of single-pass segmentation methods.The introduction identifies GPU memory as a constraint on fitting the entire process into memory.Source: §1 · p. 1
- Downsampling loses fine details, whereas independent patch processing lacks global information needed to resolve local ambiguity.These limitations affect applications requiring high-resolution segmentation output.Source: §1 · p. 1
- A single global-local fusion stage can struggle to reconcile contrasting maps when ultra-high-resolution images create a large gap between whole-image and patch scales.The introduction gives GLNet as an example of this challenge.Source: §1 · p. 2
- MagNet analyzes the image from coarse to fine across multiple scales, progressively refining the segmentation at uncertain locations.Its refinement module uses one segmentation map to refine another, while integrating global contextual cues and producing detailed output under memory constraints.Source: §1 · p. 2
2. Related Work
Related work addresses scale and context through multi-scale features, multi-stage processing, global-to-local information transfer, and segmentation refinement. The introduction to MagNet positions it as a modular framework that extends refinement across multiple scales.
Related work
- Multi-scale, multi-stage context aggregation combines multiple scale levels to aggregate different fields of view and provide more context to each pixel, through cascades, feature fusion, and dilated convolution.Examples include ICNet, RefineNet, FPN, DeepLab, PSPNet, and HRNet.Source: §2 · p. 2
- Multi-stage approaches include hierarchical field-of-view scaling and foreground-presence prediction before ultra-high-resolution patch segmentation.Source: §2 · p. 2
- Global-context transfer methods add pooled information to local features, while GLNet shares global and local information but is difficult to extend beyond two scales.The authors also report difficulty training GLNet's local network because the global branch dominates.Source: §2 · p. 2
- Refinement methods include CRFs, guided filters, iterative networks, CascadedPSP, boundary refinement, and PointRend.The authors note limitations including incremental improvement or slowness for classical methods, error propagation in cascaded refinement, and limited context or tiny-object recovery for some selective refiners; MagNet is presented as modular across any number of scales.Source: §2 · pp. 2–3
3. MagNet
MagNet processes ultra-high-resolution images through stages ordered from coarsest to finest, progressively refining segmentation maps. Each stage combines a current-scale prediction with the cumulative result from previous stages; MagNet-Fast trades scale and patch coverage for efficiency.
3. MagNet
- MagNet uses multiple stages corresponding to image scales, inspecting the input from coarsest to finest and progressively refining segmentation maps.The segmentation module can be any backbone that outputs uncertainty estimates, and the refinement module is backbone-agnostic.Source: §3 · p. 3
3.1. Multi-stage processing pipeline
- At each stage, image patches are processed at a scale-specific resolution, with finer stages using smaller patches and more windows.The scale levels span the image from its full dimensions to the largest dimensions that fit the segmentation module's memory capacity.Source: §3.1 · p. 3
3.2. Refinement module
- The refinement module combines cumulative and current-scale segmentation maps, then selects locations where the cumulative prediction is uncertain and the combined prediction is more certain.Its score map multiplies the cumulative-map uncertainty by one minus the combined-map uncertainty, then applies median blurring.Source: §3.2 · p. 4
3.3. MagNet-Fast
- MagNet-Fast reduces runtime by using fewer scales and refining only selected patches, chosen by the highest cumulative-prediction uncertainty.The number of patches it processes may be smaller than in patch processing, while it retains global context and detailed information.Source: §3.3 · pp. 4–5
4. Experiments
The experiments evaluate MagNet on three high-resolution datasets spanning urban, aerial, and medical imagery. The evaluated images range from 2 to 25 million pixels.
4. Experiments
- MagNet was evaluated on Cityscapes, DeepGlobe, and Gleason, with ablation studies conducted on Cityscapes.The paper compares MagNet with other state-of-the-art semantic segmentation frameworks.Source: §4 · p. 5
- The evaluation covers urban-view, aerial-scene, and medical-image datasets through Cityscapes, DeepGlobe, and Gleason, respectively.The dataset passage identifies these three evaluation datasets; the paper context describes their image domains.Source: §4 · p. 5
- 2 to 25 million pixels: the evaluated high-resolution images span this range and contain many details.Table 2 provides dataset details.Source: Table 2 · p. 5
4.1. Implementation details
The implementation uses a two-residual-block refinement module, trained on randomly extracted patches. Inference processes nonoverlapping patches at each scale and reports mIoU.
4.1. Implementation details
- The refinement module uses two residual blocks and maps an h×w×2C input to an h×w×C output.It is the module architecture used in all experiments.Source: §4.1 · p. 5
- During training, the refinement module is trained to refine coarse outputs on sliced images using randomly extracted patches.Training also uses rotation and horizontal and vertical flipping, SGD with momentum 0.9, weight decay 5×10^-4, and an initial learning rate of 10^-3.Source: §4.1 · p. 5
- At inference, MagNet processes nonoverlapping patches at each scale and evaluates segmentation with mean Intersection over Union (mIoU).Memory and speed benchmarks use an Intel i7 CPU and an RTX2080Ti GPU.Source: §4.1 · p. 5
4.2. Experiments on the Cityscapes dataset
On Cityscapes, MagNet progressively refines segmentation across scales and outperforms the compared refinement methods. Ablations show that intermediate scales and uncertainty-guided point selection affect performance, and improvements also hold across tested backbones.
4.2. Experiments on the Cityscapes dataset
- Cityscapes contains 2048×1024 urban-scene images; the experiments used 2,975 fine-labeled training images and 500 test images, with a 256×128 segmentation-module input.Four patch scales were considered, from 2048×1024 to 256×128; larger patches were downsampled to fit the memory constraint.Source: §4.2 · p. 5
4.2.1 Benefits of multiple scale levels
- Adding two intermediate scales improved mIoU by 4.34%, compared with about 2% for direct refinement from the lowest to highest scale.After each stage, errors decreased and segmentation masks became finer.Source: §4.2.1 · p. 5, Figure 5 · p. 6
4.2.2 Comparing segmentation approaches
- MagNet achieved the highest mIoU, while MagNet-Fast significantly outperformed SegFix and DGF at similar inference times; MagNet-Fast was faster than MagNet.DenseCRF did not improve the coarse segmentation, and PointRend was fastest but yielded only a small improvement.Source: §4.2.2 · p. 6, Figure 6 · p. 7
- On the two illustrated Cityscapes images, MagNet and MagNet-Fast refined results best; SegFix missed small objects and PointRend performed poorly because it lacked global context.The figure also shows more recognized tiny objects and better-refined boundaries with MagNet and MagNet-Fast.Source: §4.2.2 · pp. 6–7, Figure 6 · p. 7
4.2.3 Ablation study: point selection
- The best point-selection performance used both uncertainty maps with median filtering, which improved results in every tested case.MagNet performance increased with more refinement points until it stopped increasing after 216; selecting all points reduced mIoU to 66.86%. PointRend performance decreased beyond 2048 selected points and fell below its unrefined result.Source: §4.2.3 · p. 7, Figure 7 · p. 7
4.2.4 Ablation study: segmentation backbones
- MagNet significantly improved the results of both tested segmentation backbones, with gains ranging from 2% to 5%.The results are reported in Table 7.Source: §4.2.4 · p. 7, Table 7 · p. 7
4.3. DeepGlobe
On the DeepGlobe satellite-image dataset, MagNet and MagNet-Fast outperformed the other methods. The comparison used a shared FPN segmentation backbone with a ResNet-50 backbone.
4.3. DeepGlobe
- DeepGlobe contains 803 satellite images of 2448×2448 pixels, annotated with seven landscape classes; the unknown class was excluded from mIoU evaluation.The reported split comprised 455 training, 207 validation, and 142 test images.Source: §4.3 · p. 8
- Both MagNet and MagNet-Fast outperformed the other methods on DeepGlobe.The comparison used the same FPN segmentation backbone for all refinement methods in the table's final part.Source: §4.3 · p. 8
- The DeepGlobe experiments used FPN with a ResNet-50 backbone and three refinement scales: 612→1224→2448.MagNet-Fast refined the three most uncertain patches at each scale; PointRend used the same segmentation backbone and outperformed GLNet.Source: §4.3 · p. 8
4.4. Gleason
On the Gleason prostate-cancer dataset, MagNet improved PSPNet's performance by 1.7%. The evaluation used ultra-high-resolution images and labels based on majority voting among six clinical experts.
4.4. Gleason
- The Gleason task segments and grades lesions into benign, Grade 3, Grade 4, and Grade 5 on 5000×5000 histopathological images.The combined final label is based on majority voting among six clinical experts.Source: §4.4 · p. 8
- MagNet improved PSPNet performance by 1.7% on the Gleason dataset.The comparison also reports MagNet-Fast, PSPNet's winning solution, and mIoU agreement between medical experts.Source: §4.4 · p. 8, Table 9 · p. 8
- The experiments used PSPNet with a ResNet-101 backbone and four refinement scales, from 625 to 5000.MagNet-Fast used the same four scales and refined the three patches with the highest prediction uncertainty at each scale.Source: §4.4 · p. 8
5. Conclusions
The conclusion presents MagNet as a multi-scale framework for high-resolution segmentation that progressively refines output across stages. Across three challenging datasets, it reports mIoU gains over previous state-of-the-art methods.
5. Conclusions
- MagNet produced high-resolution segmentation without exploding GPU memory usage by dividing input images into patches.It considers multiple scales from coarsest to finest to avoid being too global or too local.Source: §5 · p. 8
- Each stage uses the previous stage's output as input, progressively refining the segmentation.The conclusion describes the framework as multi-scale and multi-stage.Source: §5 · p. 8
- Across three challenging high-resolution image datasets, MagNet outperformed previous state-of-the-art methods by 1% to 2% mIoU.Source: §5 · p. 8