PaperlayerView exact revision on arXiv

Source-linked AI summary

CBNet: A Composite Backbone Network Architecture for Object Detection

Tingting Liang, Xiaojie Chu, Yudong Liu, Yongtao Wang, Zhi Tang, Wei Chu, Jingdong Chen, Haibin Ling

arXiv:2107.00420v7cs.CV

TL;DR

Object detection needs stronger backbones, but designing and pre-training new ones is costly. CBNet composes multiple identical pre-trained backbones with composite connections, auxiliary supervision, and pruning. It improves performance across backbones and detector designs, reaching 59.4% box AP and 51.6% mask AP on COCO with a 6× shorter schedule than Swin-L.

  • Problem

    Designing and pre-training new backbone architectures for stronger object detection requires substantial expertise, experimentation, and computational resources.

  • Method

    CBNet groups multiple identical existing pre-trained backbones with composite connections, auxiliary supervision, and pruning, without additional pre-training.

  • Results

    CBNet improves original backbones by 3.4%∼3.5% AP and supports CNN- and Transformer-based backbones plus diverse detector heads; CB-Swin-L reaches 59.4% box AP and 51.6% mask AP with a 6× shorter schedule than Swin-L.

  • Takeaways & Limitations

    CBNet offers a more efficient and adaptable way to build high-performance object-detection backbones than simply increasing network depth and width.

Abstract

from arXiv · show

Modern top-performing object detectors depend heavily on backbone networks, whose advances bring consistent performance gains through exploring more effective network structures. In this paper, we propose a novel and flexible backbone framework, namely CBNetV2, to construct high-performance detectors using existing open-sourced pre-trained backbones under the pre-training fine-tuning paradigm. In particular, CBNetV2 architecture groups multiple identical backbones, which are connected through composite connections. Specifically, it integrates the high- and low-level features of multiple backbone networks and gradually expands the receptive field to more efficiently perform object detection. We also propose a better training strategy with assistant supervision for CBNet-based detectors. Without additional pre-training of the composite backbone, CBNetV2 can be adapted to various backbones (CNN-based vs. Transformer-based) and head designs of most mainstream detectors (one-stage vs. two-stage, anchor-based vs. anchor-free-based). Experiments provide strong evidence that, compared with simply increasing the depth and width of the network, CBNetV2 introduces a more efficient, effective, and resource-friendly way to build high-performance backbone networks. Particularly, our Dual-Swin-L achieves 59.4% box AP and 51.6% mask AP on COCO test-dev under the single-model and single-scale testing protocol, which is significantly better than the state-of-the-art result (57.7% box AP and 50.2% mask AP) achieved by Swin-L, while the training schedule is reduced by 6$\times$. With multi-scale testing, we push the current best single model result to a new record of 60.1% box AP and 52.3% mask AP without using extra training data. Code is available at https://github.com/VDIGPKU/CBNetV2.

I. INTRODUCTION

CBNet constructs stronger object-detection backbones by composing multiple identical pre-trained backbones, avoiding additional pre-training. It improves accuracy and efficiency across backbones, detector heads, and testing settings.

  • Motivation: Existing backbone design and pre-training demand substantial expertise, experimentation, and computational resources for stronger detection performance.The paper motivates exploiting existing architectures and pre-trained weights instead of developing and pre-training a new backbone.
  • CBNet architecture: CBNet connects parallel assisting and lead backbones so high- and low-level features are integrated while the receptive field progressively expands.Outputs from each assisting-backbone stage flow to parallel and lower-level stages of the succeeding backbone; lead features feed the neck and detection head.
  • Training and efficiency: CBNet uses existing open-source pre-trained backbone weights and adds auxiliary supervision plus pruning to improve training and reduce model complexity without sacrificing inference speed.The composite backbone is initialized from an individual pre-trained backbone, while assisting backbones receive supervised auxiliary detection losses.
  • CBNet variants: CBNetV2 combines Dense Higher-Level Composition, auxiliary supervision, and special pruning, outperforming the earlier CBNetV1 empirically.CBNetV1 uses adjacent-stage connections, whereas CBNetV2 is designed to exploit the composite architecture more fully.
  • Experimental evidence: 3.4%∼3.5% AP improvements over original backbones and 1.1%∼2.1% AP gains at comparable complexity show CBNet’s effectiveness and efficiency.The framework generalizes across convolutional and Transformer-based backbones.
  • Experimental evidence: 59.4% box AP and 51.6% mask AP are achieved by CB-Swin-L on COCO test-dev, surpassing Swin-L’s 57.7% box AP and 50.2% mask AP with a 6× shorter schedule.Multi-scale testing reaches 60.1% box AP and 52.3% mask AP.

II. RELATED WORK

The paper situates CBNet within object-detection pipelines, detector families, backbone scaling, and automated architecture search. It contrasts composite backbones with existing feed-forward, recurrent, and search-based approaches.

  • Object Detection: Object detectors typically use a classification-pre-trained backbone, a neck for multi-scale feature enhancement, and a head for bounding-box prediction.Detection heads are commonly organized into one-stage and two-stage branches.
  • Object Detection: One-stage detectors include YOLO, SSD, RetinaNet, NAS-FPN, and EfficientDet, while two-stage methods include Faster R-CNN, FPN, Mask R-CNN, and Cascade R-CNN.The passage presents these as the two major detector branches.
  • Composite architectures: Composite strategies include Same Level, Adjacent Higher-Level, Adjacent Lower-Level, Dense Higher-Level, and Full-connected Composition.For K = 2, colored connections represent operations such as element-wise operations, scaling, 1×1 convolution, and batch normalization.
  • Object Detection: Anchor-free detectors such as FSAF, FCOS, ATSS, and GFL provide an alternative to anchor-based approaches, alongside keypoint-based methods such as CornerNet and CenterNet.The passage links this development partly to FPN and focal loss.
  • Architecture search: Neural architecture search methods explore detector architectures using reinforcement learning, evolutionary algorithms, partial-order pruning, or gradient-based optimization.Examples include NAS-FPN, NAS-FCOS, SpineNet, SM-NAS, Auto-FPN, and OPANAS.
  • Backbone design: The paper proposes grouping existing identical backbones as a more economical alternative to designing and pre-training new detection backbones.The discussed backbone families include ResNet, ResNeXt, Res2Net, HRNet, and Swin Transformer.

C. Recurrent Convolution Neural Network

CBNet differs from recurrent CNNs and model ensembles by grouping identical backbones with unidirectional, stage-wise feature propagation rather than recurrent weight sharing or separately trained predictors. Its architecture uses assisting backbones to enhance a lead backbone, whose features feed the detector.

  • C. Recurrent Convolution Neural Network: CBNet uses unidirectional connections between parallel stages, unlike RCNN's bidirectional connections and shared weights across time steps.The comparison distinguishes CBNet's architecture from unfolded recurrent CNNs.
  • C. Recurrent Convolution Neural Network: CBNet differs from model ensembles because it groups identical backbones and enhances features within one network instead of separately training diverse models and voting over predictions.Ensembles combine outputs from independently trained predictors, whereas CBNet forwards features between backbones.
  • C. Recurrent Convolution Neural Network: CBNet groups K identical backbones, with assisting backbones B1 through BK−1 forwarding features to lead backbone BK.Each backbone contains L stages, and the lead backbone supplies the output features used for object detection.
  • C. Recurrent Convolution Neural Network: Stage-wise composite connections transform assisting-backbone features to match the spatial size required by each stage in the successor backbone.The transformed features contribute to the input of every corresponding stage in the next backbone.
  • C. Recurrent Convolution Neural Network: Only the lead backbone's output features are used for detection, while assisting backbones forward features to succeeding siblings.The backbones can use architectures such as ResNet, ResNeXt, Res2Net, and Swin Transformer, initialized from one pretrained backbone.

B. Possible Composite Strategies

CBNet defines composite connections that combine features from assisting backbones with corresponding stages in later backbones. The simplest strategy fuses same-stage features, while the architecture also distinguishes CBNetV1's AHLC-based design from CBNetV2's broader strategy set.

  • B. Possible Composite Strategies: Composite connection gl(x) accepts all stage features from an assisting backbone and outputs a feature matching the target stage's size.The paper introduces five composite strategies for this operation.
  • B. Possible Composite Strategies: Same-level composition fuses output features from corresponding stages of different backbones using a 1 × 1 convolution and batch normalization layer.This strategy is illustrated as SLC.
  • B. Possible Composite Strategies: CBNetV1 uses AHLC as its composite strategy, whereas CBNetV2 combines DHLC, auxiliary supervision, and pruning.The two versions therefore differ in both composite and training strategies.

2) Adjacent Higher-Level Composition (AHLC):

AHLC feeds higher-level features from a previous backbone into subsequent lower-level stages, following the top-down feature-enhancement principle of Feature Pyramid Networks. ALLC provides the contrasting bottom-up pathway, while DHLC and FCC extend the connections across more stages.

  • 2) Adjacent Higher-Level Composition (AHLC):: AHLC feeds the adjacent higher-level stage of a previous backbone into the subsequent stage of the next backbone.The top-down pathway introduces spatially coarser but semantically stronger features to lower-level stages.
  • 3) Adjacent Lower-Level Composition (ALLC):: ALLC instead feeds the adjacent lower-level stage of a previous backbone into the succeeding stage.It forms a bottom-up pathway complementary to AHLC.
  • 4) Dense Higher-Level Composition (DHLC):: DHLC composes features from all higher-level stages in the previous backbone and adds them to lower-level stages in the latter backbone when K = 2.This expands AHLC from one adjacent connection to dense higher-to-lower connections.
  • 5) Full-connected Composition (FCC):: FCC feeds features from all stages in previous backbones to every stage in the following backbone, resizing them by down-sampling or up-sampling as needed.It additionally includes low-to-high-level connections compared with DHLC.

C. Auxiliary Supervision

CBNet training adds auxiliary supervision to assist optimization while retaining the lead-backbone loss as the primary objective. Pruning removes stages from later backbones to reduce complexity, and CBNetV2 combines these choices with DHLC for broader detector compatibility.

  • C. Auxiliary Supervision: Auxiliary supervision adds a second detection head using assisting-backbone features, while the original lead-backbone head remains primary.The auxiliary head and neck share weights with the original head and neck.
  • C. Auxiliary Supervision: The total loss balances lead-backbone and assisting-backbone losses with weights assigned to each assisting backbone.LLead is the lead loss, LAssist is the assisting loss, and λi weights the i-th assisting loss.
  • C. Auxiliary Supervision: Auxiliary supervision is removed during inference, so it does not affect inference speed.Only the lead backbone's output features are used at inference.
  • D. Pruning Strategy: Pruning removes selected stages from the second through K-th backbones and fills them with same-stage features from the first backbone.For K = 2, the paper illustrates five pruning methods corresponding to different numbers of retained stages.
  • E. Architecture of Detection Network with CBNet: CBNetV2 combines DHLC, auxiliary supervision, and pruning, and applies to off-the-shelf detectors without architectural modifications.The experiments evaluate generality across backbones, detectors, and compatibility with other components.

A. Implementation details

The experiments evaluate CBNet on COCO using standard detection metrics, baseline settings, and comparisons with anchor-free and anchor-based detectors. CB-Res2Net101-DCN achieves strong AP with comparable or shorter training schedules.

  • COCO evaluation uses 118k training images, 5k minival images for ablations, and 20k test-dev images for state-of-the-art comparisons.
  • COCO AP averages precision across IoU thresholds from 0.5 to 0.95 at different object scales.
  • The baseline is Faster R-CNN with a ResNet50 backbone and FPN, trained with SGD for 12 epochs under a step-decay schedule.
  • State-of-the-art comparisons use multi-scale training, a longer schedule, Soft-NMS, and 1600×1400 inference inputs.
  • Object Detection: 52.8% AP is achieved by CB-Res2Net101-DCN under the anchor-free single-scale protocol.
  • Object Detection: 55.6% AP is achieved by CB-Res2Net101-DCN among anchor-based detectors, using 32 training epochs.This schedule is reported as 16× and 12× shorter than EfficientDet and YOLOv4, respectively.

2) Instance Segmentation:

CBNet improves detection and instance segmentation across CNN and Swin backbones, including gains over larger conventional backbones and shorter training schedules. The reported results support its efficiency and generality for high-performance detectors.

  • Instance Segmentation: CB-Swin-S achieves 56.3% box AP and 48.6% mask AP on COCO minival with Cascade Mask R-CNN and ImageNet-1K pre-training.These results provide +4.4% box AP and +3.6% mask AP over Swin-B with similar model size and the same protocol.
  • Instance Segmentation: 58.4% box AP and 50.7% mask AP are achieved by CB-Swin-B on COCO minival with ImageNet-22K pre-training.The passage reports 17% fewer parameters and a 3.6× shorter training schedule than Swin-L (HTC++).
  • Instance Segmentation: 59.4% box AP and 51.6% mask AP are achieved by CB-Swin-L on COCO test-dev with single-scale testing.The schedule is 6× shorter than Swin-L, and multi-scale testing reaches 60.1% box AP and 52.3% mask AP.
  • Effectiveness and Efficiency: CBNet improves CNN-based backbones by over 3.4% AP across architectures including ResNet, ResNeXt-32x4d, and Res2Net.
  • Effectiveness and Efficiency: CBNet backbones are reported as more efficient than wider and deeper versions of their corresponding single backbones.

2) Generality for Swin Transformer:

CBNet generalizes across Swin Transformer, mobile, high-resolution, pyramid, and mainstream detector configurations. Across these settings, it improves accuracy while supporting varied detector heads and favorable efficiency comparisons.

  • Generality for Swin Transformer: CB-Swin-T improves Swin-T by 3.1% box AP and 2.5% mask AP under the reported Swin training protocol.
  • Various Backbones: CBNet is evaluated with MobileNetV2, HRNet, PVT, and PVTv2 using publicly available pre-trained backbones and corresponding MMDetection settings.
  • Various Backbones: CB-MobileNetV2 improves MobileNetV2 by 3.1% AP and exceeds MobileNetV2(1.4x) by 1% AP with comparable model complexity.
  • Detector Head Designs: CBNet improves popular one-stage, two-stage, anchor-based, and anchor-free detectors by over 3% AP, while Mask R-CNN gains 2.9% AP in instance segmentation.
  • Related Comparisons: CB-CSPNet-L improves CSPNet-L by 2.6% AP and exceeds CSPNet-X by 1.1% AP in the reported YOLOX comparison.
  • Related Comparisons: CBNet outperforms DetectoRS by 0.8% AP with only 84% of its FLOPs when using the Swin-Tiny backbone.

E. Compatibility of CBNet

CBNet is compatible with deformable convolution and model ensembles, while its composite strategy materially affects accuracy. Dense high-to-low connections, especially DHLC, provide a favorable accuracy–complexity trade-off across backbones.

  • Compatibility: Deformable convolution improves CBNet by 2.3% AP∼2.7% AP, and the two techniques can be superimposed without conflict.
  • Compatibility: CBNet and model ensembling can be superimposed without conflict, so the composite detector is treated as a single model despite multiple identical backbones.
  • Composite Strategies: DHLC raises the baseline from 34.6% AP to 37.3% AP, whereas SLC provides only a slight improvement and ALLC decreases performance by 2.2% AP.
  • Composite Strategies: FCC achieves 37.4% AP but is 7% slower than DHLC, at 19.9 versus 21.4 FPS.
  • Composite Strategies: DHLC is recommended over FCC because it provides computational simplicity while achieving one of the two best reported results.
  • Composite Strategies: DHLC outperforms AHLC by 0.8% AP on ResNet101 and 1.1% AP on ResNeXt101-64x4d.
  • Composite Strategy Search: A proxy-task grid search evaluates 512 detectors over 205 GPU days and identifies a simplified DHLC strategy matching the designed DHLC at 37.3% AP.The search supports the necessity of high-to-low connections in the handcrafted design.

2) Weights for Auxiliary Supervision:

Auxiliary supervision improves CBNet detector accuracy, while pruning reduces computation with minimal accuracy loss. The weighting experiments identify effective supervision settings for different numbers of backbones.

  • 1.8% AP improvement is obtained for CB-ResNet50-K3 by setting {λ1, λ2} to {0.5, 1.0}.For K = 2, setting λ1 to 0.5 improves the baseline by 0.8% AP.
  • CBNetV2 combines dense higher-level composition, auxiliary supervision, and pruning to improve on CBNetV1.Table XV compares the two versions in terms of accuracy and complexity.
  • 10% fewer FLOPs in s3 than s4 causes only a 0.1% accuracy decrease.The pruned stage’s weights remain fixed during detector training, preserving detection accuracy.
  • Pruning fixed stages in the 2nd through K-th backbones is recommended when inference speed and memory cost are priorities.

4) Number of Backbones in CBNet:

Increasing the number of composite backbones raises accuracy while CBNet can remain more computationally efficient than deeper ResNet models. Identical-backbone grouping is also supported as the preferred configuration for CBNet’s joint-training strategy.

  • 4) Number of Backbones in CBNet:: Higher accuracy is achieved at K=2 than with ResNet152 while requiring lower computation cost.Accuracy continues to increase as the number of backbones and model complexity increase.
  • 4) Number of Backbones in CBNet:: CBNetV2 improves CBNetV1 in accuracy and complexity through dense higher-level composition, auxiliary supervision, and pruning.The comparison progressively evaluates these components when applied to CBNetV1.
  • 4) Number of Backbones in CBNet:: Identical-backbone composition outperforms diverse composition within the same backbone family.The experiments report that joint training works best for identical backbone grouping, distinguishing CBNet from diversity-based ensembles.
  • 4) Number of Backbones in CBNet:: 3.4%∼3.5% AP gains over original backbones demonstrate CBNet’s generalization across different backbone architectures and detector head designs.
  • 4) Number of Backbones in CBNet:: CBNet supports CNN-based and Transformer-based backbones, including ResNet, ResNeXt, Res2Net, and Swin-Transformer.
Loading 2107.00420v7…