PaperlayerView exact revision on arXiv

Source-linked AI summary

OCNet: Object Context Network for Scene Parsing

Yuhui Yuan, Lang Huang, Jianyuan Guo, Chao Zhang, Xilin Chen, Jingdong Wang

arXiv:1809.00916v4cs.CV

TL;DR

Semantic segmentation needs better pixel classification through context that identifies the object region associated with each pixel. OCNet defines object context from same-category pixels, approximates it with dense or efficient interlaced sparse self-attention, and combines it with multi-scale schemes; it achieves competitive performance on five benchmarks.

  • Problem

    Semantic segmentation seeks accurate pixel categories, but context schemes can include irrelevant background pixels instead of focusing explicitly on the object region containing each pixel.

  • Method

    OCNet defines object context as same-category pixels, models their relations with dense or interlaced sparse self-attention, and extends it with pyramid and atrous spatial pyramid object context.

  • Results

    OCNet achieves competitive performance on Cityscapes, ADE20K, LIP, PASCAL-Context and COCO-Stuff.

  • Takeaways & Limitations

    Coarse object-context estimation empirically outperforms PPM and ASPP on various benchmarks while emphasizing semantically related object information.

  • Takeaways & Limitations

    Estimating the dense relation matrix has quadratic computational complexity with respect to the input size, and richer context construction remains future work.

Abstract

from arXiv · show

In this paper, we address the semantic segmentation task with a new context aggregation scheme named \emph{object context}, which focuses on enhancing the role of object information. Motivated by the fact that the category of each pixel is inherited from the object it belongs to, we define the object context for each pixel as the set of pixels that belong to the same category as the given pixel in the image. We use a binary relation matrix to represent the relationship between all pixels, where the value one indicates the two selected pixels belong to the same category and zero otherwise. We propose to use a dense relation matrix to serve as a surrogate for the binary relation matrix. The dense relation matrix is capable to emphasize the contribution of object information as the relation scores tend to be larger on the object pixels than the other pixels. Considering that the dense relation matrix estimation requires quadratic computation overhead and memory consumption w.r.t. the input size, we propose an efficient interlaced sparse self-attention scheme to model the dense relations between any two of all pixels via the combination of two sparse relation matrices. To capture richer context information, we further combine our interlaced sparse self-attention scheme with the conventional multi-scale context schemes including pyramid pooling~\citep{zhao2017pyramid} and atrous spatial pyramid pooling~\citep{chen2018deeplab}. We empirically show the advantages of our approach with competitive performances on five challenging benchmarks including: Cityscapes, ADE20K, LIP, PASCAL-Context and COCO-Stuff

1 Introduction

The paper introduces object context for semantic segmentation, explicitly aggregating pixels from the same category while reducing the cost of dense self-attention. It combines this scheme with multi-scale context and reports competitive performance on five benchmarks.

  • Object context targets improved pixel classification by explicitly identifying the object region containing each pixel.
  • Conventional PPM and ASPP contexts can mix object pixels with relevant and irrelevant background pixels.
  • Object context gathers pixels belonging to the same category as a given pixel, emphasizing information needed for semantic labeling.
  • Dense relation matrices approximate binary same-category relations using feature similarities, with larger values tending to occur on object pixels.
  • Interlaced sparse self-attention approximates dense relations through two sparse relation matrices, reducing complexity compared with conventional self-attention.
  • Pyramid object context and atrous spatial pyramid object context extend the approach with richer multi-scale context.
  • OCNet achieves competitive performance on Cityscapes, ADE20K, LIP, PASCAL-Context and COCO-Stuff.

2 Related Work

Prior semantic-segmentation methods use spatially nearby, rectangular, or globally aggregated context and self-attention to capture contextual dependencies. OCNet instead models relational context with an interlaced sparse mechanism intended to lower computation cost.

  • Context mechanisms in semantic segmentation commonly use nearby spatial context or multi-scale regions to aggregate pixels.
  • PPM aggregates pixels within pyramid sub-regions, while ASPP selects surrounding pixels at different dilation rates.
  • Regular rectangular context regions may include pixels belonging to background categories.
  • Object context differs by selecting pixels from the same object category and emphasizing object pixels relevant to labeling.
  • Self-attention and non-local networks model long-range contextual information, motivating their use in vision tasks.
  • OCNet factorizes the dense relation matrix into two sparse relation matrices, distinguishing it from CGNL and RCCA through a lower-cost relational mechanism.

3 Approach

OCNet defines context by object category rather than regular spatial neighborhoods, then approximates this object context efficiently with interlaced sparse self-attention and multi-scale extensions.

  • Context representation: Dense context represents each output pixel using a weighted average of selected input representations, with relation values determining the weights.The formulation uses input representation X, output representation Z, transform functions δ and ρ, and context subsets Ii.
  • Multi-scale context: OCNet combines object context with PPM and ASPP to capture richer context information and handle objects at multiple scales.The extensions estimate object context within spatial-pyramid sub-regions or combine ASPP with object context.
  • Object context: Object context selects pixels belonging to the same object category as the given pixel.This explicitly emphasizes object pixels rather than the mixed object and background pixels gathered by conventional spatial context schemes.
  • Object context: A binary N × N relation matrix records whether each pair of pixels belongs to the same category, but estimating it directly is intractable.The dense relation matrix serves as a surrogate, assigning larger values to semantically similar pixels.
  • Interlaced sparse self-attention: The sparse relation scheme factorizes dense relations into global and local sparse matrices, reducing each pixel’s comparisons to selected interlaced subsets.Global groups collect positions with equal remainder, while local groups collect positions with equal quotient; this saves computation cost.
  • Interlaced sparse self-attention: The global and local relation modules together approximate dense relations between any two pixels, as illustrated by the two-stage permutation and self-attention process.The global stage connects spatially distant positions, and the local stage regroups originally nearby positions before another self-attention operation.

4 Experimental Results

OCNet is evaluated through component studies, efficiency comparisons, and state-of-the-art benchmarks across five semantic segmentation datasets. The experiments show that object-context modeling improves accuracy, approximates object relations, and reduces computational costs relative to several alternatives.

  • Evaluation Protocol: OCNet is evaluated on Cityscapes, ADE20K, LIP, PASCAL-Context, and COCO-Stuff, with additional experiments applying it to Mask-RCNN.The evaluation includes component comparisons, state-of-the-art comparisons, and a generalization study.
  • Ablation Study: Base-OC (ISA) outperforms PPM by 0.99%/0.61% on Cityscapes/ADE20K, respectively measured by mIoU.The comparisons use the same training and testing settings, including training iterations and batch size.
  • Ablation Study: ISA achieves comparable performance to conventional self-attention while being much more efficient.The comparison is reported alongside PPM and ASPP on Cityscapes val and ADE20K val.
  • Complexity: Base-OC (ISA) is nearly 3× faster and saves more than 88% GPU memory than DANet under the reported complexity setting.The approach also requires less GPU memory and inference time than RCCA and CGNL.
  • State-of-the-Art Comparisons: On Cityscapes, ASP-OC improves HRNetV2-W48 from 81.6% to 82.5% and outperforms ACNet.OCNet also reports competitive results of 45.40% and 45.50% on ADE20K with ResNet-101 and HRNetV2-W48, respectively.
  • State-of-the-Art Comparisons: OCNet achieves 56.2% on PASCAL-Context, 56.35% on LIP, and 40.0% on COCO-Stuff in the reported evaluations.The LIP and COCO-Stuff results are described as competitive, while the PASCAL-Context result outperforms most previous approaches.
  • Qualitative Analysis: The estimated dense relation matrix places most relation weights on same-category pixels, approximating the ground-truth object context.Qualitative visualizations also report better segmentation maps and improved boundary quality for several Cityscapes categories.
  • Application to Mask-RCNN: Adding ISA improves Mask-RCNN by approximately 1% across object detection and instance segmentation metrics under both reported schedules.With the 2× schedule, box AP/mask AP increase from 38.7/34.9 to 39.7/35.7.

5 Conclusion

OCNet enhances object information by exploiting semantic relations between pixels, using dense and interlaced sparse self-attention implementations across five semantic-segmentation benchmarks.

  • OCNet presents object context to enhance object information through semantic relations between pixels.
  • The method includes dense-relation conventional self-attention and sparse-relation interlaced sparse self-attention implementations.
  • OCNet demonstrates effectiveness on Cityscapes, ADE20K, LIP, PASCAL-Context, and COCO-Stuff.
  • The approach is also extended to Mask-RCNN to examine its advantage beyond semantic segmentation.

6 Future work

The paper identifies richer context modeling as future work, proposing object co-occurrence, shape structure, and spatial-location relations as additional information for refining segmentation predictions.

  • Future context modeling could use co-occurring object categories to refine coarse segmentation maps.The paper gives rider–bicycle co-occurrence as an example for correcting rider pixels misclassified as person.
  • Shape structure information could regularize segmentation using prior shape masks for objects under varying views.Bus shapes are cited as quadrilateral, pentagon, or hexagon depending on viewpoint.
  • Spatial-location relations between objects could refine predictions using priors such as a keyboard typically lying under a monitor.
  • The visualized dense relation combines global and local relations, while checkerboard artifacts arise from the visualization implementation.
  • OCNet visualizations include dense relation matrices on LIP and COCO-Stuff, object detection and instance segmentation comparisons on COCO, and qualitative segmentation comparisons across five benchmarks.

A. More discussions on the benefits of object context.

Object context is argued to reduce context variance by emphasizing object information, and experiments report that OCNet outperforms PPM and ASPP under fair comparison settings.

  • Object context is presented as superior to PPM and ASPP because it emphasizes object information rather than mixing object and background information.
  • Object context theoretically reduces context variance by excluding useful and irrelevant background-information variance from the context representation.
  • OCNet outperforms both PSPNet and DeepLabv3 in experiments conducted under fair comparison settings.
  • PPM context aggregates pixels within the same spatial-pyramid region, with k ∈{2, 3, 6} representing different region partitions.

C. Formulation of Permutation Matrix.

The permutation matrix formulation defines how columns of global and local relation matrices are reordered when combining the two relation stages.

  • The permutation matrix P specifies column rearrangements for the global and local relation matrices.
  • When multiplying by P or P⊤, columns are permuted according to the binary entries p_i,j and p_j,i.
  • The surrounding figure materials contrast context explanation, semantic segmentation, images without context, and segmentation without context.

D. Why the sparse relation is more efficient?

The sparse relation formulation reconstructs dense relations through two sparse matrices, but its explicit reconstructed matrix still requires O(N^2) GPU memory.

  • Sparse context computation is written as Z = (WlP^⊤WgP)X, using local and global sparse relation matrices.The dense formulation is Z = WX.

E. Intuitive example of the sparse relation scheme.

The two sparse relation matrices preserve all-to-all connectivity through direct and indirect paths between output and input positions.

  • Each output position connects with every input position either directly or through an intermediate position.For A1, A2, A3, and B1 are direct connections, while B2 and B3 are reached indirectly through B1.

F. Complexity Analysis.

The analysis derives ISA’s complexity from its global and local relation stages, with minimized complexity under balanced group dimensions; Pyramid-OC additionally aggregates contexts across four pyramid partitions.

  • Complexity Analysis: The complexity proof represents the input X, transformed features, and output-related tensors using dimensions involving HW and C.The stated input and transformed feature shapes are HW × C, with ρ(X) in R^(HW×C).
  • Complexity Analysis: ISA’s overall computation combines the costs of its global and local relation stages.The proof computes the final complexity by adding T(ISA/global) and T(ISA/local).
  • Complexity Analysis: ISA minimizes computation to O(HWC^2 + (HW)^(3/2)C) when PhPw = QhQw.The condition follows from the arithmetic mean–geometric mean inequality.
  • More details of Pyramid-OC: Pyramid-OC divides an H × W × C feature map into k × k groups for k ∈ {1, 2, 3, 6}.Object context pooling is applied separately within each group, with shared parameters within each partition.
  • More details of Pyramid-OC: It computes context maps Z1, Z2, Z3, and Z6 from the four partitions, then concatenates them into the final representation.The context maps correspond to the four pyramid scales.

I. Checkered artefact with ISA.

The paper attributes the checkerboard pattern in relation visualizations to the ISA visualization implementation and explains that its shape follows the global relation map.

  • Checkered artefact with ISA: Checkerboard artifacts in the visualized global, local, and dense relations arise from the visualization implementation.The paper refers readers to Algorithm 2 for the related visualization pseudocode.
  • Checkered artefact with ISA: The checkerboard shape matches the global relation map because selected global relation matrices are multiplied with the local relation matrix.The selected matrices correspond to pixels sharing a group with the selected pixel in the local relation stage.
  • Checkered artefact with ISA: Index permutation determines how positions are arranged during the global and local relation stages of interlaced sparse self-attention.Figure 12 illustrates the stage-specific permutations using indexed spatial positions.
Loading 1809.00916v4…