Source-linked AI summary
Attention-guided Unified Network for Panoptic Segmentation
Yanwei Li, Xinze Chen, Zheng Zhu, Lingxi Xie, Guan Huang, Dalong Du, Xingang Wang
TL;DR
Panoptic segmentation methods often separate foreground instance segmentation from background semantic parsing, overlooking their underlying relationship. AUNet unifies both branches and adds object- and pixel-level attention, achieving state-of-the-art performance on MS-COCO and Cityscapes.
Problem
Existing panoptic segmentation approaches separate foreground objects and background contents, overlooking complementary contextual cues between them.
Method
AUNet combines foreground instance and background semantic segmentation in a shared-backbone two-branch network with proposal and mask attention modules.
Results
AUNet delivers consistent accuracy gains and claims state-of-the-art PQ performance on the MS-COCO and Cityscapes benchmarks.
Takeaways & Limitations
Partitioning targets into subclasses can enable complementary information to propagate across branches and assist scene understanding.
Takeaways & Limitations
The RoIUpsample layer uses zero padding for vacant areas, producing blank background context that requires future investigation.
Abstract
from arXiv · showhide
This paper studies panoptic segmentation, a recently proposed task which segments foreground (FG) objects at the instance level as well as background (BG) contents at the semantic level. Existing methods mostly dealt with these two problems separately, but in this paper, we reveal the underlying relationship between them, in particular, FG objects provide complementary cues to assist BG understanding. Our approach, named the Attention-guided Unified Network (AUNet), is a unified framework with two branches for FG and BG segmentation simultaneously. Two sources of attentions are added to the BG branch, namely, RPN and FG segmentation mask to provide object-level and pixel-level attentions, respectively. Our approach is generalized to different backbones with consistent accuracy gain in both FG and BG segmentation, and also sets new state-of-the-arts both in the MS-COCO (46.5% PQ) and Cityscapes (59.0% PQ) benchmarks.
1. Introduction
Panoptic segmentation jointly finds foreground things at the instance level and background stuff at the semantic level. AUNet unifies these tasks and uses attention from foreground predictions to improve background understanding.
- Task motivation: Panoptic segmentation identifies foreground things as instances and background stuff semantically.Things include countable objects, while stuff includes amorphous regions such as grass, sky, and road.
- Problem: Prior methods combined separate foreground instance-segmentation and background scene-parsing models, ignoring their underlying relationship.This separation fails to borrow contextual cues between things and stuff.
- Approach: AUNet combines instance and semantic segmentation into a unified two-branch network with shared features and end-to-end optimization.The shared backbone improves computational efficiency while enabling information flow between foreground and background branches.
- Approach: AUNet uses Proposal Attention and Mask Attention modules to provide object-level and pixel-level guidance from foreground to background segmentation.Regional proposals adjust foreground/background probabilities, while foreground masks refine the boundary between things and stuff.
- Results: AUNet reports state-of-the-art panoptic quality on MS-COCO and Cityscapes and verifies joint optimization and attention modules through ablation studies.The supplied introduction identifies PQ as the metric integrating thing and stuff accuracy but does not provide benchmark values.
- Task motivation: 6.7% absolute gap separates thing and stuff pixel accuracy under the reported MS-COCO semantic evaluation.Thing instances with the same semantic class are evaluated as one category in this comparison.
2. Related Work
Earlier scene-understanding research commonly separated foreground and background tasks, while attention-based methods added contextual information. AUNet instead establishes relationships between foreground things and background stuff through coarse-to-fine attention blocks.
- Prior task separation: Traditional scene-understanding research focused separately on foreground or background targets.Object detection, instance segmentation, and semantic segmentation advanced largely as distinct task families.
- Prior task separation: Separating tasks caused missing contextual cues in instance segmentation and confusion from individuals in semantic segmentation.The cited passage ends mid-discussion but explicitly identifies these two effects of task separation.
- Attention methods: Attention-based approaches aggregated global context, class-dependent features, or long-range spatial and channel information.The related methods used channel attention, self-attention, and non-local filtering for contextual modeling.
- AUNet positioning: AUNet models the relationship between foreground things and background stuff with coarse-to-fine attention blocks.Its network structure shares features across foreground, background, and RPN branches.
3. Attention-guided Unified Network
AUNet unifies foreground instance segmentation and background semantic segmentation through shared features and cross-branch attention. Proposal- and mask-based attention provide progressively coarser-to-finer contextual guidance from foreground processing to background understanding.
- 3.2. Unified Framework: AUNet combines separate foreground and background models into parallel branches sharing an FPN backbone and jointly generating both outputs.The unified design replaces independently modeled things and stuff with shared multi-scale features.
- 3.2. Unified Framework: The shared backbone is jointly supervised by RPN, RCNN, mask, and semantic-segmentation losses during end-to-end training.The joint loss balances the four training objectives with dataset-specific coefficients.
- 3.3.1 Proposal Attention Module: Proposal Attention Module uses RPN features to highlight foreground and background regions in shared semantic feature maps.It computes foreground-weighted maps, complements them to emphasize background, and applies element-wise attention and summation.
- 3.3.1 Proposal Attention Module: A simple background reweighting function downweights less useful background layers after proposal attention.The paper explicitly notes that this reweighting function could be improved but leaves that extension beyond scope.
- 3.3.2 Mask Attention Module: Mask Attention Module uses foreground masks to refine the boundary between foreground things and background stuff at pixel level.Because fixed-shape masks encode finer layouts than RPN features, RoIUpsample restores them to spatial feature-map locations before attention aggregation.
- 3.3.2 Mask Attention Module: RoIUpsample reverses RoIAlign-like spatial processing by mapping m × m foreground masks back to feature maps using accurate RoI locations.The restored feature map is assigned to scales according to RoI size for subsequent mask attention operations.
4. Experiments
Experiments evaluate AUNet on MS-COCO and Cityscapes using PQ, then analyze unified training and attention modules through component comparisons and benchmark results.
- Datasets and metrics: MS-COCO provides 115k training images, 5k validation images, 80 thing categories, and 53 stuff categories for panoptic evaluation.Cityscapes provides 2975 training images, 500 validation images, 19 semantic labels, and 8 annotated instance categories; its coarse annotations are unused.
- Datasets and metrics: PQ combines segmentation quality (SQ) and recognition quality (RQ), with separate PQTh and PQSt results reported for things and stuff.True positives are matched segments with IoU(p, g) > 0.5; false positives and false negatives are unmatched predicted and ground-truth segments.
- Component analysis: 2.4% absolute PQ improvement over separate training demonstrates the benefit of the unified framework in component-wise MS-COCO analysis.The unified framework alone improves both PQSt and PQTh, contributing 1.1% absolute PQ improvement through shared-backbone joint optimization.
- Component analysis: PAM consistently improves PQSt and PQTh by using multi-scale FG-BG attention, while background reweighting aggregates highlighted background features with global context.Alternative feature-fusion methods had minor contributions, supporting attention as the more appropriate relationship-establishment mechanism.
- Component analysis: MAM provides finer background representations, but its contribution is slightly lower than PAM, possibly because FG masks lack contextual cues.Zero padding in RoIUpsample creates blank background context; higher-resolution masks add 0.1% absolute PQ, and MAMr reaches 39.2% PQ on MS-COCO.
- Comparison to state-of-the-arts: AUNet reaches 46.5% PQ on MS-COCO test-dev without extra data or network enhancements using a single ResNeXt-152-FPN model.It improves over PKU 360 by about 4.9% in PQSt and over JSIS-Net by 19.3% absolute PQ.
- Comparison to state-of-the-arts: On Cityscapes, AUNet surpasses the previous state of the art by 5.2% absolute PQ, while disabling MAM because it decreases PQ in that setting.Using all 19 semantic labels outperforms treating all things as one background category; the authors attribute the decline to inconsistent prior information.
5. Conclusions
AUNet unifies foreground and background segmentation while using two attention sources to let the foreground branch guide background understanding. The approach reports consistent gains and new state-of-the-art results, while motivating broader cross-subclass information sharing.
- AUNet unifies foreground instance-level and background semantic-level segmentation in one model.
- AUNet's unified design supports complementary information flow between foreground things and background stuff.
- AUNet examples on MS-COCO show performance on things exceeding human annotations, with same-class instances displayed in different color intensities.
- Proposal and mask attention provide object-level and pixel-level guidance to the background branch, respectively.
- The paper argues that partitioning visual targets into subclasses can enable complementary information to propagate across subclasses, while noting further possibilities remain unexplored.