Source-linked AI summary
Segmentation Transformer: Object-Contextual Representations for Semantic Segmentation
Yuhui Yuan, Xiaokang Chen, Xilin Chen, Jingdong Wang
TL;DR
Semantic segmentation needs context aggregation that reflects the object class associated with each pixel. The paper introduces OCR, which learns supervised object regions, aggregates their representations, and relates pixels to regions; it reports competitive performance across five benchmarks, including 84.5% on Cityscapes test.
Problem
The paper investigates context aggregation for semantic segmentation, where each pixel’s label corresponds to the object it belongs to.
Method
OCR learns soft class-specific object regions, aggregates pixel representations within each region, and augments pixels using relation-weighted object-region representations.
Results
OCR achieves competitive performance across Cityscapes, ADE20K, LIP, PASCAL-Context, and COCO-Stuff, including 84.5% on Cityscapes test.
Takeaways & Limitations
Object-contextual representations consistently improve semantic segmentation across the evaluated benchmarks.
Abstract
from arXiv · showhide
In this paper, we address the semantic segmentation problem with a focus on the context aggregation strategy. Our motivation is that the label of a pixel is the category of the object that the pixel belongs to. We present a simple yet effective approach, object-contextual representations, characterizing a pixel by exploiting the representation of the corresponding object class. First, we learn object regions under the supervision of ground-truth segmentation. Second, we compute the object region representation by aggregating the representations of the pixels lying in the object region. Last, % the representation similarity we compute the relation between each pixel and each object region and augment the representation of each pixel with the object-contextual representation which is a weighted aggregation of all the object region representations according to their relations with the pixel. We empirically demonstrate that the proposed approach achieves competitive performance on various challenging semantic segmentation benchmarks: Cityscapes, ADE20K, LIP, PASCAL-Context, and COCO-Stuff. Cityscapes, ADE20K, LIP, PASCAL-Context, and COCO-Stuff. Our submission "HRNet + OCR + SegFix" achieves 1-st place on the Cityscapes leaderboard by the time of submission. Code is available at: https://git.io/openseg and https://git.io/HRNet.OCR. We rephrase the object-contextual representation scheme using the Transformer encoder-decoder framework. The details are presented in~Section3.3.
1 Introduction
The paper targets semantic segmentation context aggregation by representing each pixel through its corresponding object region. OCR forms supervised soft object regions, aggregates region representations, and uses pixel–region relations to augment pixel features, producing improvements across five benchmarks.
- Semantic segmentation assigns a class label to every image pixel, supporting practical applications such as autonomous driving.
- Prior context methods aggregate surrounding pixels using spatial scale or representation similarity, without explicitly structuring context by object class.
- OCR learns a coarse soft region for each class under ground-truth supervision, aggregates pixels within each region, and augments each pixel with relation-weighted region representations.
- Unlike multi-scale schemes, OCR distinguishes same-object-class from different-object-class context; unlike prior relational schemes, it models pixel–region rather than only pixel–pixel relations.
- OCR achieves competitive results on Cityscapes, ADE20K, LIP, PASCAL-Context, and COCO-Stuff, including 84.5% on Cityscapes test.
2 Related Work
Related work aggregates context through multi-scale sampling, pixel relations, or region representations. OCR instead learns class-corresponding regions with segmentation supervision and computes relations using both pixel and region representations.
- Multi-scale context: Multi-scale methods such as PSPNet and DeepLab capture context through pyramid pooling, dilated convolutions, or multi-resolution features.
- Multi-scale context: ASPP samples pixels across dilation rates, causing context pixels to span object and background regions rather than a specific object.
- Relational context: Self-attention relational methods weight contextual pixel representations by pixel-to-pixel similarity.
- Relational context: Double Attention and ACFNet group pixels into regions and aggregate region representations using context relations.
- Relational context: OCR learns regions under ground-truth segmentation supervision and computes pixel–region relations from both pixel and region representations.
- OCR pipeline: The OCR pipeline forms soft object regions, estimates their representations, then computes object-contextual and augmented representations.
- Region-wise segmentation: Region-wise segmentation methods organize pixels into regions to learn better pixel representations for labeling.
3 Approach
The approach organizes image pixels into soft class-specific object regions, computes region representations, and augments each pixel using relations to those regions. The OCR pipeline can be expressed as a Transformer encoder-decoder in which category queries extract regions and cross-attention aggregates object context.
- Semantic segmentation assigns each pixel one of K class labels, motivating context aggregation that reflects object-level categories.
- OCR partitions pixels into K soft object regions, each corresponding to a class and represented by a coarse segmentation map.The regions are learned from an intermediate backbone representation under ground-truth segmentation supervision.
- Each object-region representation aggregates pixel representations weighted by their degrees of belonging to that region.Spatial softmax normalizes the region weights before aggregation.
- For each pixel, OCR computes relations to all object regions and aggregates their transformed representations into an object-contextual representation.The relation uses transformed pixel and region representations, while the contextual representation follows the weighted aggregation formulation.
- The final pixel representation fuses the original pixel representation with its object-contextual representation through a learned transform.This distinguishes OCR from methods whose regions do not correspond to object classes or whose relations use only pixel representations.
- In the Transformer formulation, decoder cross-attention uses category queries to extract soft object regions and compute their representations, while encoder cross-attention aggregates object context.The decoder self-attention unit can interact with category queries, and the encoder self-attention unit may be local and optional.
- The comparison in Table 1 reports superior performance for the approach, attributing the result to using both pixel and region representations in relation computation.The region representation characterizes the object in the specific image, supporting more accurate relations than pixel representations alone.
4 Experiments: Semantic Segmentation
Across semantic-segmentation benchmarks, OCR consistently outperforms multi-scale and relational context schemes while reducing context-module complexity. With stronger baselines and training data, it achieves competitive or leading results, including 84.5% on Cityscapes test.
- Context-scheme comparisons: OCR consistently outperforms PPM and ASPP across Cityscapes, ADE20K, and LIP comparisons.The comparison is conducted under fair settings with a dilated ResNet-101 backbone.
- Context-scheme comparisons: OCR consistently performs better than relational context schemes across different benchmarks.Double Attention’s region-number hyperparameter was fine-tuned to 64 for its best reported performance.
- Efficiency: OCR requires less memory, computation, and runtime than competing context modules, including 1/4 of PPM’s GPU memory and 1/2 of its runtime.It also uses 1/2 of PPM’s FLOPs; the comparisons exclude backbone complexity.
- Experimental setup: OCR is evaluated against multi-scale and relational context schemes using simple and advanced baselines for fair comparison.The simple baseline is dilated ResNet-101 with stride 8; the advanced baseline is HRNet-W48 with stride 4.
- State-of-the-art comparison: 84.5% on Cityscapes test is achieved by HRNet + OCR + SegFix, ranking first on the Cityscapes leaderboard at submission time.HRNet-W48 + OCR reaches 84.2%, and SegFix adds 0.3 percentage points.
- State-of-the-art comparison: OCR achieves 45.66% on ADE20K val, 56.65% on LIP val, 56.2% on PASCAL-Context test, and 40.5% on COCO-Stuff test.The reported results are competitive or best-performing depending on the benchmark and baseline.
5 Experiments: Panoptic Segmentation
The paper extends OCR to panoptic segmentation on COCO using Panoptic-FPN. The implementation reuses the semantic-segmentation OCR design without tuning to refine the semantic map.
- Task and dataset: OCR is applied to panoptic segmentation, which unifies instance and semantic segmentation, using the COCO dataset.The evaluation uses all 2017 COCO images with 80 thing classes and 53 stuff classes.
- Implementation: The semantic head’s original prediction generates soft object regions, after which an OCR head predicts a refined semantic segmentation map.The original semantic head and OCR head each receive a loss weight of 0.25.
- Implementation: The same OCR implementation is used without tuning, while other training settings remain unchanged for fair comparison.The supplied passage introduces the results section but does not provide the numerical results.
6 Conclusions
The paper presents object-contextual representations for semantic segmentation, strengthening each pixel representation with the representation of its corresponding object region and improving performance consistently across benchmarks.
- Object-contextual representations strengthen each pixel by characterizing it with the representation of its corresponding object region.
- The approach brings consistent improvements on various semantic segmentation benchmarks.