PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Skip Connections: Top-Down Modulation for Object Detection

Abhinav Shrivastava, Rahul Sukthankar, Jitendra Malik, Abhinav Gupta

arXiv:1612.06851v2cs.CVcs.LG

TL;DR

Object detection needs fine details that early convolutional layers lose, while selecting useful low-level features requires top-down context. The paper adds a top-down modulation network with lateral connections, achieving strong COCO gains across several ConvNet architectures.

  • Problem

    Object detection needs finer details for difficult categories, but early convolutional layers lose these details and skip connections lack top-down contextual selection.

  • Method

    The TDM network supplements a bottom-up ConvNet with top-down features and lateral connections that modulate, select, and integrate lower-layer features.

  • Results

    TDM improves vanilla Faster R-CNN from 23.3 AP to 28.6 AP with VGG16, from 31.5 AP to 35.2 AP with ResNet101, and achieves 37.3 AP with InceptionResNetv2 on COCO.

  • Takeaways & Limitations

    TDM features benefit objects of different sizes and support accurate localization in COCO detection experiments.

  • Takeaways & Limitations

    The endpoint criteria for how far top-down modulation should extend remain unresolved, leaving this design choice as future work.

Abstract

from arXiv · show

In recent years, we have seen tremendous progress in the field of object detection. Most of the recent improvements have been achieved by targeting deeper feedforward networks. However, many hard object categories such as bottle, remote, etc. require representation of fine details and not just coarse, semantic representations. But most of these fine details are lost in the early convolutional layers. What we need is a way to incorporate finer details from lower layers into the detection architecture. Skip connections have been proposed to combine high-level and low-level features, but we argue that selecting the right features from low-level requires top-down contextual information. Inspired by the human visual pathway, in this paper we propose top-down modulations as a way to incorporate fine details into the detection framework. Our approach supplements the standard bottom-up, feedforward ConvNet with a top-down modulation (TDM) network, connected using lateral connections. These connections are responsible for the modulation of lower layer filters, and the top-down network handles the selection and integration of contextual information and low-level features. The proposed TDM architecture provides a significant boost on the COCO testdev benchmark, achieving 28.6 AP for VGG16, 35.2 AP for ResNet101, and 37.3 for InceptionResNetv2 network, without any bells and whistles (e.g., multi-scale, iterative box refinement, etc.).

1. Introduction

The paper addresses the loss of fine details in bottom-up ConvNets by using top-down contextual information to select and integrate lower-level features for object detection.

  • Motivation: Fine edge details needed to recognize objects such as bottles and remotes are lost because early ConvNet layers learn invariance against those edges.Relevant context, such as tables, emerges only late in the feedforward architecture.
  • Motivation: Skip connections recover lower-layer details, but directly incorporating high-dimensional skip features can overfit and lacks feature selection.The paper argues that lower-level features require an attention or selection mechanism.
  • Approach: TDM supplements a bottom-up ConvNet with a top-down network and lateral connections that modulate lower-layer filters and integrate features.The top-down pathway transmits high-level semantics downward while combining them with laterally processed intermediate features.
  • Approach: The resulting representation combines local and larger receptive fields and can optionally increase feature spatial resolution.These features are used as the final detection representation.

2. Related Work

Related work combines features across ConvNet levels, uses contextual feedback, or refines representations, while TDM positions top-down and lateral connections as a complementary detection framework.

  • Skip Connections: Skip-connection methods combine finer lower-layer features with coarse semantic features through fusion, resampling, or independent multi-level predictions.The proposed framework can control upsampling, subsampling, and fusion through lateral and top-down connections.
  • Lateral Architectures: Encoder-decoder and segmentation approaches use lateral connections but generally only invert pooling and do not modulate the bottom-up network.TDM instead models the flow of high-level context features.
  • Proposal Refinement: Proposal-refinement methods select proposals from feedforward features before applying top-down refinement, limiting refinement to proposals already selected.The paper argues those feedforward features are insufficient for small and difficult objects.
  • Feature Pyramids: Feature Pyramid Networks use multiple proposal generators and region classifiers, while TDM learns one final feature map for a single proposal generator and classifier.Both approaches use bottom-up, top-down, and lateral connections, but their detection designs differ.
  • Contextual Feedback: Some prior methods use unrolled feedback networks or contextual regions and objects, whereas TDM explores a complementary top-down architecture with lateral connections.The paper states that TDM can be combined with unrolled approaches.

3. Top-Down Modulation (TDM)

TDM uses lateral modules to transform bottom-up features and top-down modules to combine them with contextual representations, producing detection features through end-to-end training.

  • Overview: TDM selects fine details from lower-level feature maps using top-down contextual features and also selects contextual features using low-level details.This bidirectional selection is the framework’s central design idea.
  • Overview: The network starts from the final bottom-up layer, while every TDM layer receives bottom-up features through lateral connections.The output captures both pertinent finer details and high-level information.
  • Modules: Each lateral module transforms a bottom-up feature, and each top-down module combines lateral and top-down features to produce a new top-down feature.These modules control the modulation network’s capacity through their output dimensions.
  • Detection Integration: The detector uses the final top-down module’s feature map, with gradient updates backpropagating through top-down, lateral, and bottom-up modules.Lateral modules transform low-level features, while top-down modules learn which semantic information and low-level details to preserve.
  • Training: The architecture study uses single-convolution modules, and training progressively adds lateral/top-down module pairs while optimizing the full network end-to-end.The modules are intentionally kept simple to analyze the impact of top-down modulation.
  • Modules: Top-down modules can optionally upsample intermediate outputs when the next lateral feature has higher spatial resolution.The final top-down module can apply a 1 × 1 convolution and ReLU before sending features to the detector.

4. Approach Details

The paper applies TDM within Faster R-CNN using ConvNet-specific base architectures and configurable top-down and lateral modules. Design choices control feature capacity, dimensionality, resolution, and compatibility with detection components.

  • 4.1. Preliminaries: Faster R-CNN: Faster R-CNN uses a shared ConvNet feature extractor, an RPN for proposing ROIs, and an RCN for classification and box regression.ROI Pooling converts each proposed region into a fixed-length representation for the RCN.
  • 4.2. Preliminaries: Base Network Architectures: The evaluated base architectures are VGG16, ResNet101, and InceptionResNetv2, with architecture-specific divisions between shared base and classifier networks.The base network is truncated at conv5_3 for VGG16, conv4_x for ResNet101, and Block 20x for InceptionResNetv2.
  • 4.3. Top-Down Modulation: TDM design varies the extent of modulation, lateral-connection frequency, and capacities of T, L, and T_out modules across architectures.The design principles assign larger capacity to coarse semantic modules and use lateral and top-down connections to reduce feature dimensionality.
  • 4.3. Top-Down Modulation: TDM is added progressively to pretrained detection models, extending top-down and lateral modules toward early features while preserving usable detector interfaces.For VGG16, modules reach conv1_x; other architectures extend to conv1 or Mixed 5b, with fixed T_out dimensions supporting pretrained RPN and RCN modules.
  • 4.3. Top-Down Modulation: The COCO evaluation compares TDM against Faster R-CNN baselines and other methods using potentially different proposal and classifier networks.The table separates published comparisons, the authors’ primary Faster R-CNN baseline, and TDM results.
  • 4.3. Top-Down Modulation: High-resolution TDM features require strided RPN processing because dense losses at finer resolutions make training prohibitively slow.The RCN can still use finer-resolution features efficiently through ROI Pooling.

5. Results

TDM is evaluated with Faster R-CNN on COCO using VGG16, ResNet101, and InceptionResNetv2, producing consistent gains across architectures and object categories. Improvements are especially pronounced for localization and small-object detection.

  • COCO Results: 28.6 AP with VGG16, 35.2 AP with ResNet101, and 37.3 AP with InceptionResNetv2 demonstrate consistent gains over corresponding Faster R-CNN baselines.The evaluation uses the COCO testdev2015 split and reports gains across network architectures.
  • COCO Results: More than 50% of VGG16 categories improve by at least 5 AP points, with similar trends for ResNet101 and InceptionResNetv2.For VGG16, TDM improves every category except one.
  • Improved localization: 8.9 AP75 points improve over the VGG16 Faster R-CNN baseline, compared with a 3.5 point AP50 boost.ResNet101 and InceptionResNetv2 obtain 4.8 AP75 and 3.1 AP75 boosts, respectively.
  • Improvement for small objects: 4.8, 3, and 3.6 APS points improve for VGG16, ResNet101, and InceptionResNetv2, respectively, on small objects.On minival*, APS boosts are 5.6, 7.4, and 8.5 points for the same architectures.
  • Qualitative Results: Qualitative results show detections of small remotes and sportsballs, including in images with heavy clutter.The examples appear in multiple rows and columns of Figure 6.

6. Design and Ablation Analysis

Ablation studies examine modulation depth, lateral modules, the top-down path, and initialization. They show that top-down modulation generally helps, lateral modules contribute beyond added depth, and endpoint selection remains unresolved.

  • Experimental protocol: Ablations use COCO trainval* for training and minival* for evaluation, while the main quantitative results also use testdev2015.The study evaluates object detection with the COCO mean average precision metric.
  • How low should the Top-Down Modulation go?: Adding more top-down modulation helps in general, but VGG16 performance saturates at T_out 2.Extending modulation to T_out 1 does not seem to help much.
  • Importance of lateral modules: Lateral modules provide a much larger performance boost than increasing network depth alone in VGG16 control baselines.The comparison controls for extra parameters by enlarging convolutional layers.
  • No top-down modules: Removing the top-down path leaves overall AP largely unchanged, although APS increases slightly in the Skip-pooling baseline.The result highlights the role of high-level contextual features in selecting low-level features.
  • Impact of pre-training: Detection-task pre-training is consistently better than classification initialization, but initialization does not change performance by a huge margin.This comparison is reported for the base networks and ResNet101/InceptionResNetv2 RPN and RCN networks.

7. Conclusion

The paper introduces TDM, which combines top-down contextual features with lateral connections to select and integrate low-level details for object detection. Experiments show benefits across object sizes and localization, while qualitative results are presented on randomly selected minival images.

  • 7. Conclusion: TDM combines top-down contextual features with lateral connections to bottom-up features, selecting and integrating low-level details for object detection.The architecture supplements a standard feedforward ConvNet with a top-down network.
  • 7. Conclusion: Qualitative TDM results are shown on randomly selected images from the minival set.
  • 7. Conclusion: TDM features benefit objects of different sizes and support accurate localization.
  • 7. Conclusion: The authors suggest that top-down modulated features may help beyond object detection across computer vision tasks.
Loading 1612.06851v2…