PaperlayerView exact revision on arXiv

Source-linked AI summary

CCNet: Criss-Cross Attention for Semantic Segmentation

Zilong Huang, Xinggang Wang, Yunchao Wei, Lichao Huang, Humphrey Shi, Wenyu Liu, Thomas S. Huang

arXiv:1811.11721v2cs.CV

TL;DR

Semantic segmentation requires contextual information beyond conventional local receptive fields, while dense full-image attention is computationally expensive. CCNet uses recurrent criss-cross attention and category consistent loss to capture full-image dependencies efficiently and produce discriminative features. It reports leading performance across several segmentation benchmarks, with measured memory and FLOP reductions versus non-local attention.

  • Problem

    Conventional FCNs provide only local contextual information, while dense non-local attention incurs O(N^2) time and space costs.

  • Method

    CCNet aggregates horizontal and vertical context with criss-cross attention, recurrently captures full-image dependencies, and applies category consistent loss.

  • Results

    CCNet achieves leading performance across Cityscapes, ADE20K, LIP, CamVid, and COCO, while RCCA uses 11× less GPU memory and reduces FLOPs by about 85% versus a non-local block.

  • Takeaways & Limitations

    Criss-cross attention provides an efficient way to capture full-image contextual information for dense prediction tasks.

  • Takeaways & Limitations

    A reported LIP example misclassifies a skirt as pants, although the paper notes the case is difficult even for humans.

Abstract

from arXiv · show

Contextual information is vital in visual understanding problems, such as semantic segmentation and object detection. We propose a Criss-Cross Network (CCNet) for obtaining full-image contextual information in a very effective and efficient way. Concretely, for each pixel, a novel criss-cross attention module harvests the contextual information of all the pixels on its criss-cross path. By taking a further recurrent operation, each pixel can finally capture the full-image dependencies. Besides, a category consistent loss is proposed to enforce the criss-cross attention module to produce more discriminative features. Overall, CCNet is with the following merits: 1) GPU memory friendly. Compared with the non-local block, the proposed recurrent criss-cross attention module requires 11x less GPU memory usage. 2) High computational efficiency. The recurrent criss-cross attention significantly reduces FLOPs by about 85% of the non-local block. 3) The state-of-the-art performance. We conduct extensive experiments on semantic segmentation benchmarks including Cityscapes, ADE20K, human parsing benchmark LIP, instance segmentation benchmark COCO, video segmentation benchmark CamVid. In particular, our CCNet achieves the mIoU scores of 81.9%, 45.76% and 55.47% on the Cityscapes test set, the ADE20K validation set and the LIP validation set respectively, which are the new state-of-the-art results. The source codes are available at \url{https://github.com/speedinghzl/CCNet}.

1 INTRODUCTION

Semantic segmentation needs long-range contextual information, but conventional and dense attention methods trade context for computational cost. CCNet addresses this with recurrent criss-cross attention and category-consistent features, reporting leading benchmark performance.

  • FCNs are limited to local receptive fields, so insufficient contextual information adversely affects segmentation accuracy.
  • Dilated convolutions gather information from few surrounding pixels, while pooling methods aggregate homogeneous context nonadaptively for every pixel.
  • Dense non-local attention captures full-image context but requires O(N^2) time and space for pixel-pair relationships.
  • Criss-cross attention uses sparse horizontal and vertical connections, and two recurrent modules let each position collect information from all image pixels.
  • Category consistent loss encourages same-category feature vectors to cluster and different-category vectors to separate, addressing potential over-smoothing.
  • CCNet achieves leading performance across Cityscapes, ADE20K, LIP, CamVid, and COCO benchmarks.
  • The extended work adds category consistent loss, 3D criss-cross attention, and experiments on LIP, CamVid, and COCO.

2 RELATED WORK

Related work develops contextual aggregation through multi-scale convolutions, pooling, attention, recurrent models, and graph-based methods. CCNet differs by combining full-image pixel access with sparse criss-cross connectivity.

  • 2.1 Semantic segmentation: Semantic segmentation methods use encoder-decoder designs, dilated convolutions, adaptive operators, graph models, adversarial learning, and real-time architectures.
  • 2.2 Context modeling: Context aggregation includes atrous spatial pyramid pooling, dense or searched multi-scale architectures, point-wise attention, adaptive pyramid context, and recurrent networks.
  • 2.3 Graph neural networks: CRF, MRF, self-attention, and non-local modules model long-range dependencies, with non-local attention generating a large spatial correlation map.
  • 2.3 Graph neural networks: Unlike GCN, both Non-local Networks and CCNet allow a pixel at any position to perceive contextual information from all pixels.
  • 2.3 Graph neural networks: The paper situates CCNet within deep graph neural network approaches for modeling contextual information in image understanding.

3 APPROACH

CCNet combines criss-cross attention, recurrent propagation, and category-consistent feature learning to capture dense contextual information efficiently for dense prediction.

  • General framework: CCNet uses a fully convolutional framework that produces feature maps, aggregates context, fuses it with local representations, and predicts segmentation results.The framework applies convolutional processing, contextual aggregation, feature fusion, and a final segmentation layer.
  • Criss-cross attention: The criss-cross attention module computes attention from reduced-dimensional Q and K features over positions sharing a row or column with each query position.Affinity scores are normalized into an attention map, while V features provide the values used for contextual aggregation.
  • Recurrent propagation: Recurrent criss-cross attention propagates information beyond each pixel’s immediate criss-cross path, enabling dense contextual information from all image pixels.With two loops, information can pass through intermediate positions to connect positions that are not directly on the same row or column.
  • Category-consistent features: The category consistent loss encourages similar features for pixels in the same category and separated features for pixels from different categories.Its discriminative formulation uses category centers, valid category elements, piece-wise distances, and variance and distance margins.
  • 3D extension: 3D Criss-Cross Attention extends the 2D module by collecting contextual information along an additional temporal or axial dimension.It operates on a feature map with temporal, width, and height dimensions and generates Q and K using 1 × 1 × 1 convolutions.

4 EXPERIMENTS

CCNet is evaluated across semantic, instance, human-parsing, and video-segmentation benchmarks, with state-of-the-art results reported on three benchmarks and gains on COCO.

  • Evaluation scope: CCNet is evaluated on Cityscapes, ADE20K, COCO, LIP, and CamVid across semantic segmentation, instance segmentation, human parsing, and video segmentation tasks.The experiments are described as comprehensive across these datasets and task settings.
  • Reported outcomes: CCNet achieves state-of-the-art performance on Cityscapes, ADE20K, and LIP, while bringing constant performance gains on COCO instance segmentation.

4.1 Datasets and Evaluation Metrics

The experiments use benchmark-specific metrics across five datasets covering urban scenes, scene parsing, human parts, instance segmentation, and autonomous-driving video.

  • Evaluation metrics: Mean IoU is used for Cityscapes, ADE20K, LIP, and CamVid, while COCO uses Average Precision.Mean IoU is defined as the mean of class-wise intersection over union.
  • Datasets: Cityscapes contains 5,000 finely annotated urban-segmentation images split into 2,975 training, 500 validation, and 1,525 testing images.
  • Datasets: ADE20K contains dense labels for 150 stuff/object categories and uses 20k, 2k, and 3k images for training, validation, and testing.
  • Datasets: LIP provides 50,462 finely annotated human-parsing images with 19 human-part labels plus background, split into 30k, 10k, and 10k images.
  • Datasets: COCO is an instance-segmentation benchmark with 115k training images across 80 categories, plus 5k validation and 20k testing images.
  • Datasets: CamVid contains 701 densely annotated 720 × 960 images from five video sequences for autonomous-driving semantic segmentation.

4.2 Implementation Details

Implementation uses task-specific backbone replacements and standard optimization settings, with Cityscapes results compared against state-of-the-art context aggregation approaches.

  • Network structure: Semantic segmentation uses ImageNet-pretrained ResNet-101 with its final two down-sampling operations removed and subsequent dilated convolutions.The resulting output stride is 8.
  • Network structure: Human parsing replaces CE2P’s Context Embedding module with RCCA, while instance segmentation uses Mask-RCNN as its baseline.
  • Training settings: Training uses SGD with mini-batches, momentum 0.9, weight decay 0.0001, and a polynomial learning-rate policy with power 0.9.For Cityscapes and ADE20K, the initial learning rate is 1e-2.
  • Comparisons: The Cityscapes test comparison is presented in Table 1, with annotations indicating extra COCO training and combined train-fine and val-fine training.

4.3 Experiments on Cityscapes

Cityscapes experiments show that recurrent criss-cross attention improves segmentation while capturing dense contextual information with substantially lower resource use than non-local attention.

  • State-of-the-art comparison: CCNet substantially outperforms prior state-of-the-art methods on the Cityscapes test set despite lower computation and memory usage than PSANet.Validation comparisons are qualified because competing methods use different training sets, testing strategies, or backbones.
  • RCCA loop ablation: 2 loops improve performance by 1.8%, while 3 loops add only 0.4% and increase FLOPs and GPU memory usage.The authors choose R = 2 to balance performance and resource usage.
  • RCCA loop ablation: R = 1 improves performance by 2.9%, demonstrating the benefit of criss-cross attention over the baseline.The authors attribute further gains from additional loops to denser contextual information.
  • Category consistent loss: CCL brings approximately 0.7% mIoU gains with both ResNet-101 and ResNet-50 on Cityscapes validation.The loss maps same-category pixels closer and different-category pixels farther apart in feature space.
  • Context aggregation comparison: RCCA outperforms non-local and other context aggregation approaches, consistent with the value of full-image contextual information.The recurrent design lets the second attention step use features containing longer-range dependencies.
  • Efficiency comparison: 11× less GPU memory and about 85% fewer FLOPs than the non-local block make RCCA a more efficient full-image context module.Recurrent non-local attention gains more than 1 point but still requires huge GPU memory usage.
  • Attention visualization: With R = 1, attention follows the target pixel’s criss-cross path; with R = 2, RCCA aggregates denser contextual information.Qualitative comparisons show challenging regions are progressively corrected as loops increase.

4.4 Experiments on ADE20K

On the challenging ADE20K scene parsing benchmark, CCNet with category consistent loss achieves state-of-the-art validation performance.

  • ADE20K results: 45.76% is the state-of-the-art ADE20K validation performance achieved by CCNet with CCL.This exceeds previous state-of-the-art methods by more than 1.1% and the conference version by 0.5%.
  • Context integration: CCNet captures full-image dependencies as an alternative to global pooling with image-level supervision for integrating contextual information.Most compared methods use ResNet-101, while RefineNet uses ResNet-152.

4.5 Experiments on LIP

On the challenging LIP human parsing benchmark, CCNet achieves state-of-the-art performance and produces accurate segmentation for complicated poses, with a documented skirt–pants confusion.

  • LIP results: 55.47% is the state-of-the-art LIP validation performance achieved by CCNet.It exceeds the previous state of the art by more than 2.3%.
  • Qualitative results: The method produces accurate segmentation for complicated poses, but one example misclassifies a skirt as pants.The paper notes that this example is difficult for humans to recognize as well.

4.6 Experiments on COCO

On COCO instance segmentation, RCCA is added to Mask R-CNN and substantially improves the baseline across reported metrics without using category consistent loss.

  • COCO instance segmentation: RCCA substantially outperforms the Mask R-CNN baseline across all reported COCO box AP and mask AP metrics.The experiment uses ResNet-50/101 backbones and omits category consistent loss for fair comparison.

4.7 Experiments on CamVid

On CamVid video semantic segmentation, CCNet uses 3D criss-cross attention to collect dense spatial and temporal context and achieves 79.1% mIoU on the test set.

  • 4.7 Experiments on CamVid: 79.1% mIoU is achieved on the CamVid test set, outperforming all other methods by a large margin.Evaluation follows the standard split of 367 training, 101 validation, and 233 test images, using single-scale scores.
  • 4.7 Experiments on CamVid: The CamVid experiment demonstrates the effectiveness of 3D-RCCA for video semantic segmentation in driving scenarios.CamVid is described as one of the first datasets focused on video semantic segmentation for driving scenarios.
  • 4.7 Experiments on CamVid: The experiment applies CNNs frame by frame, then concatenates and reshapes features for the 3D Criss-Coss Attention module.The setup uses R = 3 to collect dense spatial and temporal contextual information.

5 CONCLUSION AND FUTURE WORK

The paper concludes that CCNet and RCCA capture full-image contextual information efficiently while achieving outstanding performance across semantic and instance segmentation datasets.

  • 5 CONCLUSION AND FUTURE WORK: CCNet adaptively captures contextual information on criss-cross paths for dense prediction tasks.The conclusion introduces RCCA as the mechanism for aggregating contextual information from all pixels.
  • 5 CONCLUSION AND FUTURE WORK: RCCA captures full-image contextual information with less computation and memory cost.This conclusion states the efficiency advantage without specifying a numerical reduction.
  • 5 CONCLUSION AND FUTURE WORK: The category consistent loss is introduced to learn discriminative features.The conclusion lists this loss alongside RCCA as a component of CCNet.
  • 5 CONCLUSION AND FUTURE WORK: CCNet achieves outstanding performance consistently on Cityscapes, ADE20K, LIP, CamVid, and COCO.The cited table references identify comparisons on ADE20K, LIP, COCO, and CamVid; Cityscapes is also named in the conclusion.
  • 5 CONCLUSION AND FUTURE WORK: The source codes are released to facilitate related research and applications.The conclusion explicitly states the release of CCNet source code.
Loading 1811.11721v2…