PaperlayerView exact revision on arXiv

Source-linked AI summary

Rotate to Attend: Convolutional Triplet Attention Module

Diganta Misra, Trikay Nalamada, Ajay Uppili Arasanipalai, Qibin Hou

arXiv:2010.03045v2cs.CV

TL;DR

Existing attention mechanisms motivate lightweight ways to model channel and spatial dependencies, but dimensionality reduction can lose inter-channel correspondence and conventional channel attention can omit spatial interactions. The paper introduces triplet attention, a three-branch rotation-based module that captures cross-dimension dependencies with minimal overhead, and reports improved baseline performance across classification and detection tasks.

  • Problem

    The paper investigates how to build lightweight attention that captures cross-dimension dependencies without information bottlenecks or indirect channel-weight correspondence.

  • Method

    Triplet attention uses three parallel branches that permute input tensors, apply Z-pooling and convolution, generate sigmoid attention weights, and aggregate the branch outputs.

  • Results

    Triplet attention improves baseline ResNet and MobileNet performance on ImageNet classification and MS COCO detection while introducing minimal computational overhead.

  • Takeaways & Limitations

    The experiments support capturing dependencies across tensor dimensions when computing attention weights as an effective and cheap design choice.

  • Takeaways & Limitations

    The paper leaves evaluation on more sophisticated architectures such as EfficientNets and extension to 3D vision for future work.

Abstract

from arXiv · show

Benefiting from the capability of building inter-dependencies among channels or spatial locations, attention mechanisms have been extensively studied and broadly used in a variety of computer vision tasks recently. In this paper, we investigate light-weight but effective attention mechanisms and present triplet attention, a novel method for computing attention weights by capturing cross-dimension interaction using a three-branch structure. For an input tensor, triplet attention builds inter-dimensional dependencies by the rotation operation followed by residual transformations and encodes inter-channel and spatial information with negligible computational overhead. Our method is simple as well as efficient and can be easily plugged into classic backbone networks as an add-on module. We demonstrate the effectiveness of our method on various challenging tasks including image classification on ImageNet-1k and object detection on MSCOCO and PASCAL VOC datasets. Furthermore, we provide extensive in-sight into the performance of triplet attention by visually inspecting the GradCAM and GradCAM++ results. The empirical evaluation of our method supports our intuition on the importance of capturing dependencies across dimensions when computing attention weights. Code for this paper can be publicly accessed at https://github.com/LandskapeAI/triplet-attention

1. Introduction

The paper targets attention mechanisms that are lightweight yet capture dependencies across channel and spatial dimensions. It proposes triplet attention as an efficient plug-in module and reports gains on classification with minimal overhead.

  • 1. Introduction: Attention mechanisms improve feature representations by learning where to attend and focusing on target objects.The paper motivates attention as a way to model channel dependencies or spatial masks in convolutional networks.
  • 1. Introduction: Triplet attention captures cross-dimension interaction through a three-branch structure designed for efficient attention computation.The branches model interactions between channel and spatial dimensions while preserving the input tensor shape.
  • 1. Introduction: The method builds attention weights without dimensionality reduction, avoiding indirect correspondence between channels and weights.This distinguishes it from prior channel-attention mechanisms that reduce dimensionality.
  • 1. Introduction: 2.28% Top-1 accuracy improvement is achieved on ResNet-50 while adding 4.8K parameters and 4.7e-2 GFLOPs.The baseline ResNet-50 has 25.557M parameters and 4.122 GFLOPs.
  • 1. Introduction: The method is evaluated on ImageNet-1k classification and object detection on PASCAL VOC and MS COCO, with Grad-CAM visualizations providing additional insight.The paper presents triplet attention as a plug-in module for classic backbone networks.

2. Related Work

Prior attention methods efficiently model channel or spatial information but can lose spatial information, incur complexity, or omit cross-dimension interaction. Triplet attention addresses these limitations by explicitly modeling cross-dimension dependencies with low overhead.

  • 2. Related Work: SENet efficiently computes channel attention, while CBAM and BAM combine spatial attention with channel attention.These methods were developed to improve feature representations at relatively low computational cost.
  • 2. Related Work: Residual Attention Network directly generates three-dimensional attention maps but is computationally complex compared with newer attention methods.The paper contrasts this complexity with more efficient attention computation approaches.
  • 2. Related Work: Figure 2 compares SE, CBAM, GC, and triplet attention using feature dimensions such as C × H × W and operations including matrix multiplication and element-wise multiplication.The figure labels the four module designs as separate comparison panels.
  • 2. Related Work: Triplet attention targets cross-dimension interaction while avoiding unnecessary dimensionality reduction and maintaining negligible computational overhead.This distinguishes it from the described prior methods that do not account for cross-dimension interaction.

3. Proposed Method

Triplet attention is a lightweight three-branch module that captures dependencies across channel and spatial dimensions without dimensionality reduction. It uses tensor rotations, Z-pooling, convolutional transformations, and averaged branch outputs to refine feature tensors with low overhead.

  • Motivation: Triplet attention targets cheap channel and spatial attention while avoiding the dimensionality reduction used by CBAM and SENet.Its design emphasizes cross-dimension interaction and is nearly parameter-free.
  • Motivation: Global pooling can remove spatial information from channel descriptors, while separately computed channel and spatial attention omits their relationship.Triplet attention is motivated by modeling these cross-dimension dependencies directly.
  • Building Blocks: Z-pool concatenates average- and max-pooled features across one tensor dimension, reducing that dimension to two while preserving a richer representation.For an input of shape (C × H × W), Z-pool produces (2 × H × W).
  • Triplet Attention: The module has three parallel branches: two rotate the input to connect C with H or W, and the third captures spatial dependencies between H and W.Each branch generates attention weights through Z-pool, convolution, batch normalization, and sigmoid activation.

4. Experiments

Experiments evaluate triplet attention across ImageNet classification, object detection, branch ablations, and Grad-CAM visualizations, showing improved performance with minimal overhead.

  • ImageNet: Triplet attention matches or outperforms similar attention techniques while adding the fewest model parameters.The comparison covers standard architectures and attention mechanisms on ImageNet.
  • ImageNet: 2.04% improvement in top-1 error rate is achieved on ImageNet with ResNet-50, alongside approximately 0.02% more parameters and ≈1% more FLOPs.The reported comparison uses a ResNet50-based model augmented with triplet attention.
  • MS COCO: Adding triplet attention improves COCO AP by over 2 points across architectures while using the same ImageNet backbone with negligible computational overhead.The experiments cover Faster R-CNN, Mask R-CNN, and RetinaNet.
  • Ablation Study on Branches: All three branches perform consistently better than the vanilla version and two branch-disabled counterparts in the CIFAR-10 ablation.The ablation separately disables the spatial branch or the two channel-related branches.
  • Visualization: Grad-CAM and Grad-CAM++ visualizations show triplet attention capturing tighter and more relevant image bounds than the compared models.The visualizations use sample images and colored overlays based on gradients of the top-class prediction.

5. Conclusion

The conclusion presents triplet attention as an efficient, information-preserving layer that improves baseline architectures across classification and detection tasks. It also identifies broader architectures and 3D vision as future directions.

  • Conclusion: Triplet attention captures feature importance across tensor dimensions using an efficient computation method without information bottlenecks.The method is presented as a novel attention layer.
  • Conclusion: Triplet attention improves ResNet and MobileNet baseline performance on ImageNet classification and MS COCO object detection with minimal computational overhead.The conclusion states the cross-architecture and cross-task outcome directly.
  • Future Work: The authors expect other cross-dimension dependency techniques may improve the results while reducing cost.They also plan to investigate EfficientNets and extend the approach to 3D vision.

A. Supplementary Experiments

Supplementary experiments extend evaluation to keypoint detection, instance segmentation, kernel-size effects, and additional Grad-CAM visualizations.

  • Supplementary Experiments: The supplementary section evaluates triplet attention on vision tasks adjacent to the paper’s main classification and detection focus.These experiments are described as additional performance evaluations.
  • Supplementary Experiments: The experiments add a keypoint detection head to Mask R-CNN and evaluate Mask R-CNN on COCO instance segmentation.They also examine convolution kernel size within triplet attention across standard architectures.
  • Supplementary Experiments: Additional Grad-CAM and Grad-CAM++ visualizations are provided to examine patterns in the resulting heatmaps.The patterns are discussed further in the supplementary visualization section.

B. Effect of kernel size k

Kernel-size experiments on CIFAR-10 and ImageNet generally show better performance with larger kernels, except that MobileNetV2 favors a smaller kernel.

  • Effect of kernel size k: Increasing kernel size generally improves performance across the evaluated CIFAR-10 and ImageNet CNN architectures.The trend is reported across experiments using different network architectures.
  • Effect of kernel size k: MobileNetV2 is an exception, with a smaller kernel outperforming the general larger-kernel trend in lighter-weight models.The kernel-size comparison is summarized for standard CNN architectures.

C. GradCAM

GradCAM and GradCAM++ visualizations compare vanilla ResNet-50, CBAM, and triplet attention on ImageNet images. The examples suggest triplet attention can produce tighter or wider, more meaningful heatmaps and can support predictions requiring class-specific or global context.

  • Visualization quality: Triplet attention generated heatmaps that were consistently tighter or wider when required and more meaningful across additional examples.The visualizations followed the same GradCAM and GradCAM++ procedure used in the paper on ImageNet test images.
  • Class-specific attention: In the first example, triplet attention correctly predicted TapePlayer, whereas CBAM and vanilla ResNet-50 predicted iPod.The image contains a cassette-player-like device and an iPod, while ImageNet assigns the TapePlayer label.
  • Global context: The second example suggests that correctly predicting power drill requires global context because few local features identify that class.All models focused on a similar image region, but CBAM and vanilla ResNet-50 made incorrect predictions with reasonably high confidence.

D. COCO Instance Segmentation

The study augments Mask R-CNN with triplet attention for COCO instance segmentation and compares its AP scores with models trained under similar schemes. The reported table emphasizes higher performance gain with minimal computational overhead.

  • Model and evaluation: Triplet attention augments Mask R-CNN for instance segmentation on COCO 2017, producing segmentation masks alongside bounding boxes.The model uses the detectron2 code base and is evaluated with various AP scores against other models using similar training schemes.
  • Reported outcome: Table 2 reports higher instance-segmentation performance gain for triplet attention with minimal computational overhead.The comparison is presented as instance-segmentation mAP (%) on MS-COCO.

E. COCO Keypoint Detection

The paper extends Mask R-CNN to COCO human keypoint detection using the same training schedule as the segmentation and detection experiments. The reported comparisons show improvement over the vanilla architecture and competitive or consistently higher performance across the listed metrics.

  • Experimental setup: Mask R-CNN is further trained on the COCO human keypoint detection task using a configuration similar to the instance-segmentation and object-detection models.The keypoint head generates 1,500 proposals per image through the Faster R-CNN region proposal network.
  • Keypoint results: Triplet attention improves over the vanilla architecture and achieves competitive results relative to the more complex CBAM model for person keypoint detection.This comparison is summarized in the Table 3 caption.
  • Bounding-box results: Triplet attention produces consistently higher gains across all reported object-detection metrics when bounding boxes are generated during keypoint training.These results are reported as mAP (%) on the MS COCO validation set using Keypoint R-CNN.
  • Baselines: The keypoint detector is compared with a vanilla baseline and CBAM, which is described as computationally more expensive despite obtaining similar results.Table 3 reports AP scores for keypoint annotations on the COCO 2017 validation set.
Loading 2010.03045v2…