PaperlayerView exact revision on arXiv

Source-linked AI summary

Attention-guided Unified Network for Panoptic Segmentation

Yanwei Li, Xinze Chen, Zheng Zhu, Lingxi Xie, Guan Huang, Dalong Du, Xingang Wang

arXiv:1812.03904v2cs.CV

TL;DR

Panoptic segmentation combines foreground instance segmentation with background semantic segmentation, but existing approaches often treated them separately. AUNet unifies both branches with proposal- and mask-based attention, achieving state-of-the-art PQ on MS-COCO and Cityscapes.

  • Problem

    Existing panoptic approaches often treated foreground instance segmentation and background scene parsing separately, leaving their complementary contextual relationship unmodeled.

    Source: §1 · p. 1

  • Method

    AUNet jointly models foreground instance and background semantic segmentation in two branches, linking them with proposal- and mask-based object- and pixel-level attention.

    Source: §1 · p. 2, §5 · p. 8

  • Results

    AUNet achieves state-of-the-art PQ on MS-COCO and Cityscapes, with consistent accuracy gains reported in MS-COCO.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The authors conclude that partitioning targets by properties can support cross-subclass information sharing for scene understanding, while broader visual-task applications remain unexplored.

    Source: §5 · p. 8

    Zero-padding vacant RoIUpsample areas produces blank background context, which the authors say needs future investigation.

    Source: §4.2.3 · p. 7

Abstract

from arXiv · show

This paper studies panoptic segmentation, a recently proposed task which segments foreground (FG) objects at the instance level as well as background (BG) contents at the semantic level. Existing methods mostly dealt with these two problems separately, but in this paper, we reveal the underlying relationship between them, in particular, FG objects provide complementary cues to assist BG understanding. Our approach, named the Attention-guided Unified Network (AUNet), is a unified framework with two branches for FG and BG segmentation simultaneously. Two sources of attentions are added to the BG branch, namely, RPN and FG segmentation mask to provide object-level and pixel-level attentions, respectively. Our approach is generalized to different backbones with consistent accuracy gain in both FG and BG segmentation, and also sets new state-of-the-arts both in the MS-COCO (46.5% PQ) and Cityscapes (59.0% PQ) benchmarks.

1. Introduction

Panoptic segmentation identifies foreground things at the instance level and background stuff at the semantic level; AUNet unifies both tasks and routes foreground cues to support background segmentation.

1. Introduction

  • Earlier approaches combined instance-segmentation models with scene-parsing algorithms [45], overlooking relationships and contextual cues between things and stuff.Panoptic segmentation [23] evaluates foreground things at the instance level and background stuff at the semantic level; the benchmark algorithm [23] and MS-COCO challenge winners used this separate-task approach.Source: §1 · p. 1
  • Combining instance- and semantic-segmentation branches in one network improves accuracy and computational efficiency through a shared backbone.The authors attribute the gains to complementary information from foreground objects and background contents.Source: §1 · p. 1
  • PAM uses regional proposals for object-level attention, while MAM uses foreground masks for pixel-level boundary refinement; both connect the foreground branch to the background branch.The authors report that foreground segmentation is often more accurate; on the MS-COCO semantic metric, thing pixel accuracy was 6.7% higher than stuff accuracy when instances of a semantic class were grouped together. Background refinement also feeds back through gradients, and both segmentation accuracies improve.Source: §1 · p. 2
  • AUNet is end-to-end and backbone-agnostic, and the paper claims state-of-the-art PQ on MS-COCO [28] and Cityscapes [8].The introduction also says an ablation study verifies the benefits of joint optimization and both attention modules.Source: §1 · p. 2
  • The paper presents AUNet as an initial exploration of complementary information between foreground objects and background contents, with potential relevance to a wider range of visual tasks.The authors identify panoptic segmentation as a natural setting for studying this relationship.Source: §1 · p. 2

2. Related Work

Related work established panoptic segmentation to unite foreground instance and background semantic segmentation, while AUNet links these branches through coarse-to-fine attention blocks.

Panoptic Segmentation

  • Prior panoptic methods combined instance and semantic models [23], used weak supervision, or fused foreground and background cues end-to-end [11]; AUNet establishes their relationship in an end-to-end framework.The section also notes recent work [22] [29] [40].Source: §2 · p. 2

Instance Segmentation

  • Instance segmentation methods are proposal-based, with accurate regional proposals and examples including MNC [9], FCIS [25], Mask RCNN, and PANet [31], or segmentation-based, aggregating pixel cues with semantics [2] [26] [32] or depth ordering [44].Source: §2 · p. 2

Semantic Segmentation

  • Semantic segmentation advanced with encoding-decoding networks such as FCN [33], while context methods include multi-scale ASPP in DeepLab [6], global-context DenseASPP, and contextual priors in PSPNet [45].Source: §2 · p. 2

Attention-based Modules

  • Attention modules have been used across visual tasks [7] [19] [37] [46] [47]; prior scene-understanding work models global context and class-dependent features [42] [43] or long-range spatial and channel context [12], while AUNet uses coarse-to-fine attention blocks to relate things and stuff.Source: §2 · pp. 2–3

3. Attention-guided Unified Network

AUNet unifies foreground things and background stuff segmentation in parallel branches that share features, using foreground proposals and masks to guide background understanding. Its proposal- and mask-based attention modules model complementary relationships between the branches.

3.1. Problem and Baselines

  • Separate thing and stuff models with similar backbones form the baseline for the unified framework.The baseline produces foreground things and background stuff results separately.Source: §3.1 · p. 3

3.2. Unified Framework

  • AUNet shares an FPN backbone across parallel foreground and background branches, jointly optimizes both tasks, and adds proposal- and mask-derived attention to background features.The foreground branch uses proposal-based instance segmentation, while lightweight background heads aggregate shared multi-scale features.Source: §3.2 · pp. 3–4

3.3. Attention-guided Modules

  • PAM and MAM connect foreground cues to the background branch, linking proposal-based and FCN-based segmentation through spatial attention.The modules draw complementary contextual cues from foreground things to support background stuff understanding.Source: §3.3 · p. 4

3.3.1 Proposal Attention Module

  • PAM uses RPN features to highlight background regions in shared features and reweight them, while also enhancing activated foreground weights during backpropagation.Its background attention map is formed as 1 − sigmoid of the foreground weighted map.Source: §3.3.1 · p. 4

3.3.2 Mask Attention Module

  • MAM uses RoIUpsample to reconstruct spatial foreground features from fixed-shape masks, then applies pixel-level attention to guide background regions.The foreground masks also receive semantic-loss supervision, and RoIUpsample is designed as an inverse of RoIAlign.Source: §3.3.2 · pp. 4–5, Figure 4 · p. 5, Figure 5 · p. 5

3.4. Implementation Details

  • Training jointly optimizes AUNet’s losses, and inference resolves overlapping thing segments before combining them with stuff results.The implementation uses ResNet-FPN backbones; loss weights differ by dataset, and final results use multi-scale testing for MS-COCO but not Cityscapes.Source: §3.4 · pp. 5–6

4. Experiments

AUNet is evaluated on MS-COCO and Cityscapes, with component analyses tracing gains to a unified FG/BG framework and attention modules. Its final results lead on MS-COCO and surpass the previous Cityscapes state of the art.

4.1. Dataset and Metrics

  • Experiments use MS-COCO and Cityscapes, report panoptic quality (PQ) and its SQ and RQ components, and train without extra data on MS-COCO.MS-COCO results use the validation and test-dev sets; Cityscapes results use validation with 19 semantic labels and 8 annotated instance categories.Source: §4.1 · p. 6

4.2. Component-wise Analysis and Diagnosis

  • 2.4%: AUNet’s unified approach improves PQ over separate training in single-model MS-COCO component analyses with ResNet-50-FPN and no extra data.The analyses follow the established merging method without training or inference tricks or multi-scale augmentation.Source: §4.2 · p. 6

4.2.1 Unified Framework

  • 1.1%: The unified framework improves PQ by boosting both PQSt and PQTh through a shared backbone and joint optimization.The paper reports reduced stuff misclassification and more detailed thing predictions.Source: §4.2.1 · pp. 6–7

4.2.2 Proposal Attention Module

  • PAM links things and stuff across scales, yielding consistent PQSt and PQTh gains; its background reweighting uses global context to emphasize relevant background features.Other tested fusion strategies, including concatenation and direct summation after feature transformation, contributed less than attention.Source: §4.2.2 · p. 7

4.2.3 Mask Attention Module

  • 0.1%: Higher-resolution masks improve PQ over smaller masks, and MAMr reaches 39.2% PQ with background reweighting.MAM provides finer background representations but contributes slightly less than PAM; zero padding in RoIUpsample leaves blank background context.Source: §4.2.3 · p. 7

4.3. Comparison to State-of-the-arts

  • 4.9% and 19.3%: AUNet improves PQSt over PKU 360 and overall PQ over JSIS-Net, respectively, on MS-COCO.The comparisons filter out gains from model ensembling when comparing with PKU 360; JSIS-Net is described as the former end-to-end method.Source: §4.3 · p. 7
  • 46.5% PQ: AUNet leads on MS-COCO test-dev using a single unified model without extra data or additional network enhancements.The model uses a ResNeXt-152-FPN backbone.Source: §4.3 · p. 7, Table 2 · p. 8
  • 5.2%: AUNet surpasses the previous Cityscapes state of the art; the final setup uses all 19 semantic labels and disables MAM.Treating all things as one background category (Oursequ) is inferior to using all 19 labels; the authors attribute MAM’s decline to potentially misleading cues from weaker thing predictions.Source: Table 3 · p. 8, §4.3 · p. 8

5. Conclusions

AUNet unifies foreground instance and background semantic segmentation, using foreground-derived attention to guide background understanding. The paper reports consistent MS-COCO accuracy gains and argues that propagating complementary information across target subclasses can aid scene understanding.

Conclusions

  • AUNet unifies foreground instance-level and background semantic-level segmentation, enabling the foreground branch to assist the background branch.Proposal attention and mask attention provide object-level and pixel-level guidance, respectively.Source: §5 · p. 8
  • AUNet achieves consistent accuracy gains in MS-COCO and new state-of-the-arts, as reported by the authors.The conclusion gives no specific metric values or benchmark comparison details.Source: §5 · p. 8
  • Partitioning visual targets into subclasses can let complementary information propagate across them to assist scene understanding.The paper identifies foreground things and background stuff as a natural panoptic-segmentation partition, while noting that further possibilities remain unexplored.Source: §5 · p. 8

Related papers

Loading 1812.03904v2…