PaperlayerView exact revision on arXiv

Source-linked AI summary

Collaborative Global-Local Networks for Memory-Efficient Segmentation of Ultra-High Resolution Images

Wuyang Chen, Ziyu Jiang, Zhangyang Wang, Kexin Cui, Xiaoning Qian

arXiv:1905.06368v3cs.CV

TL;DR

Ultra-high-resolution segmentation is constrained by GPU memory and by the loss of detail or context from processing images globally or locally alone. GLNet combines downsampled global images with cropped local patches through deep feature sharing, and its coarse-to-fine variant addresses class imbalance. Across ultra-high-resolution aerial and medical datasets, the approach reports high-quality segmentation with a competitive accuracy–memory trade-off, using less than 2GB for inference.

  • Problem

    Ultra-high-resolution segmentation must balance dense-prediction quality against GPU-memory limits, while downsampling or patch cropping discards global context or local detail.

  • Method

    GLNet uses global and local branches for downsampled images and cropped patches, deeply sharing features; a coarse-to-fine variant addresses class imbalance.

  • Results

    Less than 2GB of GPU memory is required for inference on ultra-high-resolution images up to 30M pixels while yielding high-quality segmentation.

  • Takeaways & Limitations

    Integrating global context with local high-resolution structures provides a memory-efficient route to high-quality ultra-high-resolution segmentation.

Abstract

from arXiv · show

Segmentation of ultra-high resolution images is increasingly demanded, yet poses significant challenges for algorithm efficiency, in particular considering the (GPU) memory limits. Current approaches either downsample an ultra-high resolution image or crop it into small patches for separate processing. In either way, the loss of local fine details or global contextual information results in limited segmentation accuracy. We propose collaborative Global-Local Networks (GLNet) to effectively preserve both global and local information in a highly memory-efficient manner. GLNet is composed of a global branch and a local branch, taking the downsampled entire image and its cropped local patches as respective inputs. For segmentation, GLNet deeply fuses feature maps from two branches, capturing both the high-resolution fine structures from zoomed-in local patches and the contextual dependency from the downsampled input. To further resolve the potential class imbalance problem between background and foreground regions, we present a coarse-to-fine variant of GLNet, also being memory-efficient. Extensive experiments and analyses have been performed on three real-world ultra-high aerial and medical image datasets (resolution up to 30 million pixels). With only one single 1080Ti GPU and less than 2GB memory used, our GLNet yields high-quality segmentation results and achieves much more competitive accuracy-memory usage trade-offs compared to state-of-the-arts.

1. Introduction

Ultra-high-resolution segmentation must balance dense-prediction accuracy with GPU memory limits. GLNet addresses this trade-off by combining global context with local detail while retaining memory efficiency.

  • 1. Introduction: Ultra-high-resolution segmentation creates a pressing trade-off between segmentation quality and GPU-memory feasibility.Dense prediction becomes substantially more memory-intensive as image resolution increases.
  • 1. Introduction: Downsampling loses high-resolution details, whereas patch cropping loses spatial context and can reduce segmentation accuracy.Experiments show downsampled predictions suffer artifacts and inaccurate boundaries, while cropped patches produce large misclassified areas.
  • 1. Introduction: 5GB to 10GB of GPU memory is required by FCN-8s and SegNet to segment one 6M-pixel ultra-high-resolution image.ICNet uses less memory but sacrifices accuracy, illustrating the accuracy–memory trade-off.
  • 1.1. Our Contributions: GLNet combines a downsampled global image with cropped local patches and deeply shares features between the two branches.The design integrates global context and local high-resolution structures during training and inference.
  • 1.1. Our Contributions: Less than 2GB of GPU memory is required for inference on images up to 30M pixels, using one 1080Ti GPU for training.The model is designed to improve the accuracy–memory usage trade-off for ultra-high-resolution segmentation.
  • 1.1. Our Contributions: The coarse-to-fine GLNet variant targets foreground–background class imbalance while keeping computation cost low.This addresses cases where foreground regions occupy little of an ultra-high-resolution image.

2. Related Work

Related work spans standard segmentation architectures, efficient models, multi-scale feature integration, and context aggregation. The paper evaluates ultra-high-resolution segmentation on public aerial and medical datasets with distinct resolutions and annotation settings.

  • 2. Related Work: FCN, U-Net, DeconvNet, SegNet, and DeepLab established encoder-decoder, skip-connection, and enlarged-receptive-field approaches for semantic segmentation.CRFs were also used to model spatial relationships.
  • 2. Related Work: ENet and ICNet reduce computation or memory through early downsampling, multi-resolution branches, and model compression.These efficient models were not customized for or evaluated on ultra-high-resolution images.
  • 2. Related Work: Multi-scale methods combine features across granularities through refinement blocks, pyramids, top-down aggregation, or staged zooming.Examples include RefineNet, Laplacian pyramids, FPN, and HAZN.
  • 2. Related Work: Context-aggregation methods encode local or non-local information using global pooling, dilated convolution, and atrous spatial pyramid pooling.These methods enlarge receptive fields while retaining feature-map resolution more effectively than simple downsampling.
  • 2.3. Ultra-high Resolution Segmentation Datasets: DeepGlobe contains 1,146 satellite images of 2448×2448 pixels with seven-class pixel masks for rural land-cover segmentation.Its classes include urban, agriculture, rangeland, forest, water, barren, and unknown.
  • 2.3. Ultra-high Resolution Segmentation Datasets: ISIC includes 2,594 dermoscopy images with binary lesion masks, and over 64% of images have ultra-high resolutions.The largest image measures 6682×4401 pixels.
  • 2.3. Ultra-high Resolution Segmentation Datasets: Inria Aerial provides 180 5000×5000-pixel images from five cities with binary building/non-building masks and city-based train/test splits.The city-based split differs from DeepGlobe’s random-tile splitting.

3. Collaborative Global-Local Networks

GLNet combines downsampled global images with cropped local patches to preserve contextual information and fine structures under limited GPU memory. Its branches exchange features deeply, aggregate their predictions, and support a coarse-to-fine variant for foreground-background imbalance.

  • Motivation: Downsampling loses high-resolution details, whereas patch cropping loses spatial context and can produce artifacts or inaccurate boundaries.These limitations motivate collaborative global-local processing.
  • Global and Local Branches: The global branch processes downsampled images, while the local branch processes fully cropped patches at higher resolution.Both branches use the same convolutional-block backbone and support training and inference over complete patch sets.
  • Deep Feature Map Sharing: Feature maps are exchanged across layers, with global maps cropped and upsampled for local patches and local maps downsampled for global-context integration.Sharing can be unidirectional or bidirectional and fuses contextual and fine-structural features.
  • Branch Aggregation with Regularization: High-level maps from both branches are concatenated and passed through a 3×3 aggregation layer to produce the final segmentation output.Auxiliary losses supervise each branch, while the main loss supervises the aggregated result.
  • Branch Aggregation with Regularization: A weakly coupled Euclidean-norm regularization synchronizes local and global learning by slowing local-branch training.The regularization is applied to the last-layer feature maps and updates only the local branch, with λ fixed at 0.15.
  • Coarse-to-Fine GLNet: The coarse-to-fine GLNet first segments downsampled images globally, then refines only a foreground-centered bounding box with the local branch.Processing beyond the bounding box as background reduces computation and addresses patches dominated by background pixels.

4. Experiments

The experiments evaluate GLNet on aerial and medical ultra-high-resolution datasets, comparing segmentation quality and memory efficiency while including ablation studies.

  • Experimental Setup: Experiments evaluate GLNet on DeepGlobe and Inria Aerial datasets and the coarse-to-fine variant on ISIC.The study compares segmentation quality, memory efficiency, and ablates design choices.
  • Experimental Setup: Bounding-box size is dynamically relaxed so the coarse-to-fine stage has a foreground-background ratio around 1.This adjustment is used to improve class balance for the second-stage local processing.

4.1. Implementation Details

Implementation uses an FPN with ResNet50, single-GPU memory measurement at batch size 1, and Adam training on NVIDIA 1080Ti hardware.

  • Model Configuration: GLNet uses an FPN with ResNet50, applying feature sharing through ResNet conv2–conv5 and FPN stages.Feature-map regularization and aggregation are used at the final lateral-connection stage.
  • Memory Measurement: GPU memory is measured with gpustat using minibatch size 1 without gradient calculation on a single GPU.The measurement protocol covers both training and inference settings described in the experiments.
  • Optimization: Training uses Adam with learning rates of 1×10^-4 for the global branch and 2 × 10^-5 for the local branch.All experiments use minibatch size 6 on NVIDIA 1080Ti GPUs.

4.2. DeepGlobe

On DeepGlobe, progressively deeper and bidirectional feature sharing improves GLNet’s mIoU, while comparisons show a favorable accuracy-memory trade-off relative to global-only and patch-only processing.

  • Dataset: DeepGlobe contains 803 images at 2448×2448 pixels with seven annotated landscape classes, excluding the unknown class from evaluation.The split includes 455 training, 207 validation, and 142 testing images.
  • Feature Sharing Ablation: 57.3% and 66.4% mIoU are achieved by global-only and local-only branches, respectively, while aggregation and regularization reach 70.3%.These values come from the local DeepGlobe test-set ablation.
  • Feature Sharing Ablation: 71.6% mIoU is achieved with bidirectional deep feature sharing, compared with 70.9% for deep sharing and a 0.2% increase from shallow sharing.The result supports progressively richer collaboration between the two branches.
  • Memory Usage: Bidirectional deep sharing increases inference memory only slightly, from 1189MB to 1865MB.The ablation also reports improved collaboration from deep sharing, aggregation, and regularization.
  • Qualitative Results: Global-only and local-only predictions show grid-like artifacts or inaccurate boundaries, while bidirectional deep sharing progressively reduces misclassification and boundary errors.Figure 7 compares aggregation, shallow sharing, and bidirectional deep sharing visually.
  • Accuracy and Memory Comparison: Global inference gives higher mIoU but uses much more GPU memory, whereas patch inference lowers memory and accuracy; GLNet achieves the best trade-off.The comparison is made on the local DeepGlobe test set across fixed image and patch sizes.
  • Image and Patch Size Comparison: FCN-8s and ICNet exhibit an accuracy-memory trade-off across image and patch sizes, showing that typical models do not balance both objectives well.FCN-8s is selected for high mIoU and ICNet for efficient memory usage.

4.3. ISIC6

On the imbalanced ISIC dataset, coarse-to-fine segmentation balances foreground and background regions, while bidirectional global-local sharing achieves 75.2% mIoU.

  • The ISIC dataset contains 2,594 ultra-high resolution images, split into 2,074 training, 260 validation, and 260 testing images.
  • 72.7% and 48.5% mIoU were achieved by the global and local branches, respectively, before coarse-to-fine refinement.
  • Coarse-to-fine refinement produces a more balanced foreground-background pixel ratio by cropping a relaxed global-branch bounding box.The local branch is trained on smaller class-balanced images, while cropped margins are assumed to be background.
  • 73.9% mIoU results from global-to-local sharing, increasing to 75.2% with bidirectional sharing on class-balanced images.The bidirectional configuration is reported as quantitatively better than other methods on both accuracy and memory usage.
  • The ISIC challenge score is defined per image as zero below 0.65 IoU and equal to IoU otherwise.

4.4. Inria Aerial

On 5000×5000-pixel Inria Aerial images, GLNet achieves 71.2% mIoU while preserving low memory usage and outperforming other methods on both accuracy and memory usage.

  • The Inria Aerial Challenge dataset contains 180 ultra-high resolution images, each measuring 5000×5000 pixels.
  • The dataset is randomly split into 126 training, 27 validation, and 27 testing images.
  • 71.2% mIoU is achieved by GLNet on the Inria Aerial testing set.
  • GLNet is quantitatively better than other methods on both accuracy and memory usage, including for 5000×5000-pixel images.The reported result preserves low memory usage for this “super” ultra-high resolution setting.

5. Conclusions

The paper concludes that GLNet combines global context and local fine structure for memory-efficient ultra-high-resolution segmentation, while coarse-to-fine processing addresses class imbalance.

  • GLNet leverages global context and local fine structure to enhance ultra-high-resolution segmentation without sacrificing GPU memory usage.
  • The coarse-to-fine segmentation approach solves the class imbalance problem identified for ultra-high-resolution images.
  • The authors position balancing GPU memory and accuracy as essential for ultra-high-resolution image analysis and describe this work as pioneering memory-efficient segmentation.
Loading 1905.06368v3…