PaperlayerView exact revision on arXiv

Source-linked AI summary

High-Resolution Deep Image Matting

Haichao Yu, Ning Xu, Zilong Huang, Yuqian Zhou, Humphrey Shi

arXiv:2009.06613v2cs.CV

TL;DR

Whole-image deep-learning matting can be impractical for high-resolution inputs because of hardware limits, while down-sampling and independent patch processing can lose detail or consistency. HDMatt uses patch-based inference and trimap-guided cross-patch context modeling. It reports state-of-the-art results on AIM and AlphaMatting benchmarks and visual results on real-world high-resolution images.

  • Problem

    Whole-image deep-learning matting can exceed hardware limits on high-resolution inputs, while down-sampling loses detail and trivial cropping causes patch-wise inconsistency.

    Source: §1 · p. 1

  • Method

    HDMatt performs patch-based matting and uses CPC, including trimap-guided TGNL, to model long-range context across patches.

    Source: §1 · pp. 1–2

  • Results

    HDMatt reports new state-of-the-art performance on AIM, AlphaMatting, and a collected real-world high-resolution dataset.

    Source: §1 · p. 2, §5 · p. 7

  • Takeaways & Limitations

    The authors state that HDMatt makes high-quality high-resolution matting practical under hardware-resource constraints.

    Source: §1 · p. 2

Abstract

from arXiv · show

Image matting is a key technique for image and video editing and composition. Conventionally, deep learning approaches take the whole input image and an associated trimap to infer the alpha matte using convolutional neural networks. Such approaches set state-of-the-arts in image matting; however, they may fail in real-world matting applications due to hardware limitations, since real-world input images for matting are mostly of very high resolution. In this paper, we propose HDMatt, a first deep learning based image matting approach for high-resolution inputs. More concretely, HDMatt runs matting in a patch-based crop-and-stitch manner for high-resolution inputs with a novel module design to address the contextual dependency and consistency issues between different patches. Compared with vanilla patch-based inference which computes each patch independently, we explicitly model the cross-patch contextual dependency with a newly-proposed Cross-Patch Contextual module (CPC) guided by the given trimap. Extensive experiments demonstrate the effectiveness of the proposed method and its necessity for high-resolution inputs. Our HDMatt approach also sets new state-of-the-art performance on Adobe Image Matting and AlphaMatting benchmarks and produce impressive visual results on more real-world high-resolution images.

1 Introduction

Deep-learning matting methods typically process whole images, but high-resolution inputs challenge hardware limits and can lose detail or consistency when down-sampled or cropped. HDMatt uses patch-based inference with cross-patch context and trimap-guided information propagation, and reports state-of-the-art results on matting benchmarks and real-world high-resolution images.

1 Introduction

  • High-resolution images can exceed hardware capacity for whole-image inference; down-sampling loses fine detail, while trivial patch cropping causes inconsistency.Large or fully unknown regions within patches also require long-range contextual information.Source: §1 · p. 1
  • HDMatt crops high-resolution images into patches and uses CPC to model long-range context across patches for each query patch.CPC samples correlated reference patches and ensembles their features for the query patch.Source: §1 · pp. 1–2
  • The authors describe HDMatt as the first deep-learning approach to high-resolution image matting, intended to make high-quality results practical under hardware constraints.The contribution is framed for real-world use.Source: §1 · p. 2
  • CPC incorporates TGNL, which uses trimap labels to compare unknown query pixels separately with foreground, background, and unknown regions in reference patches.This guides information propagation across different pixel types.Source: §1 · p. 2
  • HDMatt achieves new state-of-the-art image-matting performance on AIM, AlphaMatting, and a newly collected real-world high-resolution dataset.The paper reports both quantitative and qualitative results.Source: §1 · p. 2

2 Related Work

Prior matting work spans sampling, propagation, and deep-learning approaches, while non-local operations provide a way to model long-range context. HDMatt adapts cross-patch context modeling to matting and uses trimap guidance.

2.1 Image Matting

  • Earlier matting approaches include sampling matched foreground and background pixels and propagation methods that optimize alpha mattes using local smoothness.Non-local matting also samples matching pixels as context for unknown-pixel prediction.Source: §2.1 · p. 2
  • Unlike Divide-and-Conquer’s RGBXY-nearby pixels, HDMatt samples context patches in high-level feature space to capture long-range semantic context.The authors describe this as a distinct sampling strategy for patch-based high-resolution matting.Source: §2.1 · p. 2
  • Deep-learning matting advanced with larger datasets and models such as DIM, AlphaGan, ContextNet, IndexNet, and GCAMatting, but ultra-high-resolution inference remains constrained by hardware.The paper positions its patch-based method and cross-patch modeling as addressing this inference setting.Source: §2.1 · p. 2

2.2 Non-local Operations

  • The overview depicts query and context patches and trimaps entering a shared encoder, followed by CPC and a decoder that estimates the query patch’s alpha.Green and red boxes denote training query and context patches; yellow boxes denote consecutive test queries.Source: Figure 2 · p. 3
  • Non-local operations model long-range dependencies across vision and language tasks; HDMatt adapts them for cross-patch context and trimap guidance.The paper calls the resulting modules CPC and TGNL.Source: §2.2 · pp. 2–3

3 The Proposed HDMatt Approach

HDMatt handles high-resolution matting by cropping images and trimaps into patches, using CPC to transfer cross-patch information, and stitching patch alpha estimates into a full matte.

3.1 Patch Cropping and Stitching

  • Inference uses overlapping image and trimap patches, with padding beyond image boundaries and blending across overlaps for smooth transitions.Training samples patches of different sizes and resizes them to 320 × 320.Source: §3.1 · p. 3

3.2 Cross-Patch Context Module

  • For each query, HDMatt ranks context patches by feature correlation with its unknown region and propagates from only the top K; empirically, K=3 achieves comparable accuracy to using all patches.Source: Context Patch Sampling · pp. 3–4
  • TGNL compares the query’s unknown-region features separately with context unknown, foreground, and background regions, then concatenates the three propagated feature maps for decoding.The operation still works when context trimaps lack one or more of the three regions, and the model is empirically robust in those cases.Source: Trimap-Guided Non-Local (TGNL) Operation · p. 4

3.3 Network Structure and Losses

  • The network uses a ResNet-34 and ASPP encoder with an unpooling decoder, trained end-to-end on equal alpha and composite losses weighted at patch boundaries by the blending mask.Source: §3.3 · p. 4

4 Experiments

HDMatt is evaluated on standard matting benchmarks and real-world high-resolution images, with ablations testing its context design and patch choices. Results report strong benchmark performance and finer real-world details, while attention visualizations show the use of distant context patches.

4.1 Dataset

  • Training uses AIM’s 431 annotated foreground images, with synthetic composites and randomly dilated trimaps; training patches are sampled from unknown regions.Patches with side lengths 640, 480, or 320 are resized to 320 × 320 and randomly flipped or rotated.Source: §4.1 · pp. 4–5

4.2 Implementation

  • The model uses a two-stage training strategy: ImageNet pretraining of a ResNet-34 encoder, followed by end-to-end training of the encoder-decoder with CPC.The default context count is K = 3; ultra-high-resolution inputs use N = 30 candidate context patches.Source: §4.2 · p. 5

4.3 Adobe Image Matting Benchmark

  • On AIM, HDMatt performs among the best methods across SAD, MSE, Gradient, and Connectivity; naive patching worsens prior methods versus whole-image inference.The patch-wise comparisons use the same patch size and blending function as HDMatt.Source: §4.3 · p. 5

4.4 AlphaMatting Benchmark

  • On AlphaMatting, HDMatt ranks first among published methods for SAD, MSE, and Gradient, despite the benchmark images being around 800×600.The table caption identifies the top-four methods and reports HDMatt’s best overall SAD performance.Source: §4.4 · p. 5, Table 2 · p. 6

4.5 Real-world Images

  • On real-world HR images, HDMatt produces finer, more accurate details and faster inference than CPU-tested ContextNet and IndexNet, but misses some finest details.The authors suggest AIM lacks similar examples of such small details; the tested images reach resolutions up to 6000 × 6000.Source: §4.5 · p. 6, Figure 5 · p. 6

4.6 Attention Visualization on Context Patches

  • CPC selects top-three context patches, including ones far from the query; attention maps indicate it leverages information from similar pixels in those patches.The visualization marks the query patch and sampled pixel, with brighter attention-map colors representing larger weights.Source: §4.6 · p. 6, Figure 7 · p. 7

4.7 Ablation Study

  • Ablations show CPC improves the shared-backbone baseline, TGNL improves on ordinary Non-local, patch-size SAD varies by at most 0.7, and all-context sampling reaches SAD 32.2.Performance is stable for K ≥ 3, so the experiments use K = 3 as a computation–performance trade-off.Source: Module Selection · p. 6, Patch Size · p. 7, Context Patch Number · p. 7

5 Conclusion

HDMatt uses patch-based training and inference to address hardware limits on high-resolution matting, while CPC models long-range context across patches. The paper reports state-of-the-art performance on AIM and AlphaMatting, with visual results on real-world high-resolution images.

5 Conclusion

  • HDMatt is presented as the first deep-learning model for high-resolution image matting.Its patch-based training and inference strategy is intended to overcome hardware limitations during high-resolution inference.Source: §5 · p. 7
  • CPC explicitly models long-range context dependencies across patches to maintain high-quality alpha mattes.Source: §5 · p. 7
  • The authors report new state-of-the-art results on AIM and AlphaMatting, plus impressive visual results on real-world high-resolution images.Source: §5 · p. 7

A Patch Stitching Method

The patch-stitching method combines overlapping alpha patches with blending and pixel weights to produce the whole matte.

A Patch Stitching Method

  • Predicted alpha patches are stitched with overlap, and blending merges overlapping regions for smooth transitions.Source: A Patch Stitching Method · p. 9
  • Each patch pixel receives a weight; pixels in the non-overlapping center are fixed at 1, while overlap-region weights lie in [0, 1].Source: A Patch Stitching Method · p. 9
  • For whole-matte pixels covered by multiple patches, the final alpha is a weighted sum using corresponding patch-pixel indices.Source: A Patch Stitching Method · p. 9

B The AlphaMatting benchmark

The section presents visual results for top-ranking methods on the AlphaMatting benchmark, with the figure comparing input images, trimaps, and method outputs.

The AlphaMatting benchmark

Related papers

Loading 2009.06613v2…