PaperlayerView exact revision on arXiv

Source-linked AI summary

Conformer: Local Features Coupling Global Representations for Visual Recognition

Zhiliang Peng, Wei Huang, Shanzhi Gu, Lingxi Xie, Yaowei Wang, Jianbin Jiao, Qixiang Ye

arXiv:2105.03889v1cs.CV

TL;DR

CNNs capture local features but struggle with global representations, while visual transformers model long-distance dependencies yet can deteriorate local details. Conformer combines concurrent CNN and transformer branches through Feature Coupling Units that interactively fuse both representations. Under comparable parameter complexity, it outperforms CNNs and visual transformers and shows potential as a general backbone network.

  • Problem

    CNNs have difficulty capturing global representations, whereas visual transformers can deteriorate local feature details.

  • Method

    Conformer uses concurrent CNN and transformer branches connected by Feature Coupling Units that align and interactively fuse local features with global representations.

  • Results

    Conformer outperforms both conventional CNNs and visual transformers under comparable parameters and computation budgets, including downstream tasks.

  • Takeaways & Limitations

    Conformer shows potential as a simple yet effective general backbone network for visual recognition and downstream tasks.

  • Takeaways & Limitations

    Existing global-cue approaches face trade-offs involving lower spatial resolution or deteriorated local details, while Conformer’s positional-embedding assumption is evaluated in the reported ablation.

Abstract

from arXiv · show

Within Convolutional Neural Network (CNN), the convolution operations are good at extracting local features but experience difficulty to capture global representations. Within visual transformer, the cascaded self-attention modules can capture long-distance feature dependencies but unfortunately deteriorate local feature details. In this paper, we propose a hybrid network structure, termed Conformer, to take advantage of convolutional operations and self-attention mechanisms for enhanced representation learning. Conformer roots in the Feature Coupling Unit (FCU), which fuses local features and global representations under different resolutions in an interactive fashion. Conformer adopts a concurrent structure so that local features and global representations are retained to the maximum extent. Experiments show that Conformer, under the comparable parameter complexity, outperforms the visual transformer (DeiT-B) by 2.3% on ImageNet. On MSCOCO, it outperforms ResNet-101 by 3.7% and 3.6% mAPs for object detection and instance segmentation, respectively, demonstrating the great potential to be a general backbone network. Code is available at https://github.com/pengzhiliang/Conformer.

1. Introduction

Conformer addresses the complementary weaknesses of CNNs and visual transformers by coupling local convolutional features with global transformer representations. Its concurrent dual-branch design and Feature Coupling Units preserve and interactively fuse both types of information.

  • Motivation: CNNs extract hierarchical local features but struggle with long-distance relationships important for high-level visual tasks.Expanding receptive fields can require intensive pooling that damages spatial resolution.
  • Motivation: Visual transformers capture long-distance dependencies through self-attention but can ignore local details, reducing foreground-background discriminability.The paper attributes this weakness to deteriorated local feature details.
  • Conformer: Conformer combines a CNN branch and a transformer branch to couple local features with global representations during enhanced representation learning.The branches follow ResNet- and ViT-style designs and are jointly supervised during training.
  • Conformer: The Feature Coupling Unit aligns channels, resolutions, and feature values while progressively reducing semantic divergence between the branches.It uses 1×1 convolution, down/up sampling, LayerNorm, and BatchNorm, and is inserted into successive blocks.
  • Evidence: Conformer’s coupling helps transformer features retain local details and improves their discriminability, while its CNN branch activates more complete object extents than conventional CNNs.The paper illustrates fuller object coverage and stronger foreground-background separation in visualizations.
  • Evidence: Under comparable parameter complexity, Conformer outperforms CNNs and visual transformers and shows potential as a general backbone network.This is presented as a broad comparison rather than a result tied to one task.

2. Related Work

Prior approaches add global cues to CNNs through larger receptive fields or attention, but each can sacrifice spatial resolution or local detail. Conformer instead uses a concurrent interactive fusion structure that retains both local and global representations.

  • CNNs with Global Cues: CNNs provide hierarchical local features but generally have difficulty capturing global cues.This limitation motivates methods that introduce broader context.
  • CNNs with Global Cues: Larger receptive fields, deeper architectures, pooling, dilation, deformable sampling, and global pooling offer several ways to add global context to CNNs.These methods modify spatial aggregation or reweight features to incorporate broader information.
  • CNNs with Global Cues: Global attention mechanisms capture long-distance dependencies and have also been integrated into convolutional architectures.Examples include non-local operations, attention-augmented convolutions, and object attention modules.
  • Limitations: Existing global-cue solutions can require intensive pooling that lowers spatial resolution or improperly fuse attention with convolution, deteriorating local details.The trade-off depends on whether context is added through receptive-field expansion or attention-based fusion.
  • Visual Transformers: Visual transformers use self-attention to model long-distance dependencies but often ignore local feature details.This limitation is described across visual-transformer applications and motivates subsequent improvements.
  • Conformer: Conformer is presented as the first concurrent network structure that fuses CNN and transformer features interactively.The structure is intended to inherit both architectures’ structural advantages while retaining local and global representation capabilities.

3. Conformer

Conformer is a concurrent hybrid network that preserves CNN local features and transformer global representations while interactively coupling them through FCUs. Its branches exchange features across resolutions and progressively reduce their semantic gap.

  • Overview: Global context is fed from the transformer branch to CNN feature maps, while CNN local features are fed back to transformer patch embeddings.This bidirectional exchange constitutes the interaction between the two branches.
  • Overview: Conformer concurrently combines CNN and transformer branches to preserve local features and global representations.The branches follow ResNet and ViT designs and are connected by FCUs.
  • Feature Coupling Unit: FCU aligns channel dimensions, feature resolutions, and feature values before fusing CNN feature maps with transformer patch embeddings.It uses 1×1 convolution, down/up sampling, LayerNorm, and BatchNorm for alignment.
  • Feature Coupling Unit: FCU is inserted into each block except the first to progressively reduce the semantic gap between local convolutional features and globally aggregated patch embeddings.The two feature types originate from different operators and capture different feature levels.
  • Structure Analysis: Conformer’s residual abstraction supports CNN, visual-transformer, and hybrid arrangements of bottlenecks and transformer blocks at different depths.The proposed residual structure outperforms the tested alternative sub-structures.

4. Experiments

Experiments evaluate Conformer on ImageNet classification, MSCOCO detection and segmentation, ablations, ensemble comparisons, and rotation and scale generalization. Across these settings, Conformer improves accuracy under comparable complexity and shows stronger invariance.

  • Image Classification: Conformer-B outperforms DeiT-B by 2.3% on ImageNet with comparable parameters and moderate MAC cost.Conformer-S also exceeds ResNet-152 by 4.1% and DeiT-B by 1.6%.
  • Object Detection and Instance Segmentation: 3.7% higher mAP is achieved by Conformer-S/32 than ResNet-101 for object detection under comparable parameters and GFLOPs.Conformer-S/32 uses 55.4 M parameters and 288.4 GFLOPs versus the FPN baseline’s 60.5 M and 295.7 GFLOPs.
  • Object Detection and Instance Segmentation: 3.6% higher mAP is achieved by Conformer-S/32 than ResNet-101 for instance segmentation under comparable parameters and GFLOPs.Conformer-S/32 uses 58.1M parameters and 341.4 GFLOPs versus the Mask R-CNN baseline’s 63.2 M and 348.8 GFLOPs.
  • Object Detection and Instance Segmentation: Conformer-S/32 has comparable parameters, GFLOPs, and inference speed to ResNet-101 while outperforming it on detection and instance segmentation.The comparison combines the performance and FPS evaluations.
  • Ablation Studies: Increasing either CNN or transformer branch parameters improves accuracy, while additional CNN parameters provide greater improvement with lower computational overhead.Conformer-S/32 also outperforms a serial ResNet–DeiT hybrid under comparable computational cost.

5. Conclusion

Conformer combines CNN-based local feature extraction with transformer-based global representation learning through an interactive Feature Coupling Unit. Experiments report gains over CNNs and visual transformers under comparable budgets and potential as a general backbone.

  • Conclusion: Conformer uses convolution operators for local features, self-attention for global representations, and FCU to fuse them interactively.The dual backbone retains both representation types in a concurrent structure.
  • Conclusion: With comparable parameters and computation budgets, Conformer outperforms conventional CNNs and visual transformers and shows potential as a general backbone network.The paper reports this across the evaluated tasks.

A. Model Architectures

Conformer-Ti, Conformer-S, and Conformer-B are model variants created by tuning the CNN and transformer branch parameters. Conformer-Ti narrows the CNN branch, while Conformer-B expands several branch dimensions.

  • Model Variants: Conformer-Ti, Conformer-S, and Conformer-B are formed by tuning parameters of the CNN and transformer branches.The variants differ in branch widths, depths, embedding dimensions, and attention heads.
  • Model Variants: Compared with Conformer-S, Conformer-Ti reduces CNN channel number by 1/4, while Conformer-B increases CNN channels, attention heads, and transformer embedding dimensions by 1.5.These changes define the reported Ti and B architecture variants.

B. Attention-based Sampling

The attention-based sampling design couples CNN feature maps with transformer patch embeddings across resolutions. It uses learned projections and reuses attention weights for downsampling and upsampling.

  • Attention-based Sampling: Conformer downsamples and upsamples between feature maps and patch embeddings using cross attention.The paper also compares this approach with pooling, convolution, and attention-based sampling strategies.
  • Attention-based Sampling: Feature maps are split into K patches, and 1×1 convolution aligns each patch’s channel dimension with transformer embedding dimension E.Each CNN patch initially has dimension n × c and becomes n × E after alignment.
  • Attention-based Sampling: For downsampling, each CNN patch is fused with a transformer patch through learned query, key, and value projections.Wq, Wk, and Wv map transformer patch embeddings to Q, K, and V.
  • Attention-based Sampling: For upsampling, the method reuses the attention weights from the downsampling equation.This reuses the established cross-resolution correspondence in the reverse direction.

C. Inference Time

Conformer is evaluated for inference speed against competing methods in classification, object detection, and instance segmentation. Conformer-S/32 has comparable efficiency to ResNet-101 while achieving better performance on the two downstream tasks.

  • C. Inference Time: Inference speed is evaluated using throughput for classification and FPS for object detection and instance segmentation.FPS is measured on a 32GB V100 GPU with batchsize 1.
  • C. Inference Time: Conformer-S/32 has comparable parameters, GFLOPs, and inference speed to ResNet-101 while outperforming it on object detection and instance segmentation.The comparison supports Conformer’s potential as a general backbone network.

D. Residual Structure

The authors examine alternative residual structures obtained by abstracting Feature Coupling Units as short connections. The proposed residual structure outperforms the tested sub-structures.

  • D. Residual Structure: Conformer can degenerate into different sub-structures under different residual connections.The alternatives are derived by treating Feature Coupling Units as short connections in the dual network structure.
  • D. Residual Structure: The proposed residual structure outperforms the other tested Conformer sub-structures.The comparison is reported in Table 10.

E. Fusion Interval

Conformer’s Feature Coupling Unit interacts local features and global representations at repeated intervals to progressively align them. Experiments indicate that more frequent fusion improves representation learning.

  • E. Fusion Interval: Feature Coupling Units interact local features and global representations in each block to progressively align features and fill the semantic gap.The fusion-interval experiment tests whether interaction should occur in every block.
  • E. Fusion Interval: Smaller fusion intervals achieve higher ImageNet performance, indicating that frequent interaction facilitates representation learning.The tested intervals perform fusion every 1, 2, or 4 blocks.

F. Convergence speed

Conformer’s convolutional operations are associated with faster early training for both its CNN and transformer branches. During the first 50 epochs, both branches significantly outperform DeiT in training accuracy.

  • F. Convergence speed: During the first 50 epochs, both Conformer-S branches significantly outperform DeiT in training accuracy.The comparison is shown on the validation set in Fig. 7.
  • F. Convergence speed: The authors attribute the faster convergence of the visual transformer to the inductive bias introduced by convolution operations.The comparison concerns both the CNN branch and the transformer branch of Conformer-S.
  • F. Convergence speed: CNNTransformer-Ti and CNNTransformer-B use multi-head self-attention configurations with 6 and 9 heads, respectively.Table 12 also reports feature-map and embedded-patch output sizes for the architectures.
Loading 2105.03889v1…