PaperlayerView exact revision on arXiv

Source-linked AI summary

CARAFE: Content-Aware ReAssembly of FEatures

Jiaqi Wang, Kai Chen, Rui Xu, Ziwei Liu, Chen Change Loy, Dahua Lin

arXiv:1905.02188v3cs.CV

TL;DR

Feature upsampling is important in modern convolutional architectures, yet conventional operators have limited contextual coverage or fixed kernels, while adaptive alternatives can be expensive. CARAFE predicts location-specific reassembly kernels from content and reassembles nearby features. Across object detection, instance segmentation, semantic segmentation, and inpainting, it reports consistent gains with little computational overhead.

  • Problem

    Feature upsampling operators must capture richer context for dense prediction while avoiding fixed, content-agnostic kernels and heavy computation.

  • Method

    CARAFE predicts content-aware reassembly kernels for each location and uses them to combine features within a predefined nearby region.

  • Results

    CARAFE consistently improves object detection, instance segmentation, semantic segmentation, and image inpainting benchmarks, including gains of 1.2% AP, 1.3% AP, 1.8% mIoU, and 1.1 dB, respectively.

  • Takeaways & Limitations

    CARAFE is presented as a universal, lightweight upsampling operator that can be integrated into modern network architectures.

Abstract

from arXiv · show

Feature upsampling is a key operation in a number of modern convolutional network architectures, e.g. feature pyramids. Its design is critical for dense prediction tasks such as object detection and semantic/instance segmentation. In this work, we propose Content-Aware ReAssembly of FEatures (CARAFE), a universal, lightweight and highly effective operator to fulfill this goal. CARAFE has several appealing properties: (1) Large field of view. Unlike previous works (e.g. bilinear interpolation) that only exploit sub-pixel neighborhood, CARAFE can aggregate contextual information within a large receptive field. (2) Content-aware handling. Instead of using a fixed kernel for all samples (e.g. deconvolution), CARAFE enables instance-specific content-aware handling, which generates adaptive kernels on-the-fly. (3) Lightweight and fast to compute. CARAFE introduces little computational overhead and can be readily integrated into modern network architectures. We conduct comprehensive evaluations on standard benchmarks in object detection, instance/semantic segmentation and inpainting. CARAFE shows consistent and substantial gains across all the tasks (1.2%, 1.3%, 1.8%, 1.1db respectively) with negligible computational overhead. It has great potential to serve as a strong building block for future research. It has great potential to serve as a strong building block for future research. Code and models are available at https://github.com/open-mmlab/mmdetection.

1. Introduction

Feature upsampling is central to dense prediction and feature-fusion architectures, but common operators trade contextual coverage, content adaptivity, and efficiency. CARAFE addresses these limitations with content-aware reassembly and delivers gains across multiple dense prediction tasks with low overhead.

  • Feature upsampling supports dense prediction decoders and feature fusion in architectures such as FPN and U-Net.
  • Nearest-neighbor and bilinear interpolation use only sub-pixel neighborhoods, while deconvolution applies instance-agnostic kernels and becomes computationally heavy with large kernels.
  • CARAFE reassembles features within a predefined region using content-aware weights generated for each location, then rearranges the results into a spatial block.
  • CARAFE predicts spatially adaptive weights on-the-fly with a lightweight fully convolutional module and softmax activation.
  • 1.2% AP, 1.3% AP, 1.8% mIoU, and 1.1 dB improvements were reported for object detection, instance segmentation, semantic segmentation, and image inpainting, respectively.For a 2× upsampling of an H × W feature map with 256 channels, CARAFE adds H ∗W ∗199k FLOPs versus H ∗W ∗1180k FLOPs for deconvolution.

2. Related Work

Prior upsampling methods include fixed interpolations and several learnable operators, but their contextual range or computational cost can limit adaptive feature upsampling. CARAFE applies lightweight content-aware feature reassembly across visual perception tasks.

  • Nearest-neighbor and bilinear interpolation use hand-crafted kernels based on spatial distances between pixels.
  • Learnable alternatives include deconvolution, Pixel Shuffle, and guided upsampling, which use different mechanisms to modify feature resolution.
  • These methods either exploit contextual information in a small neighborhood or require expensive computation for adaptive interpolation.
  • CARAFE demonstrates content-aware feature reassembly as a lightweight solution for upsampling in several visual perception tasks.

3. Content-Aware ReAssembly of FEatures

CARAFE is a content-aware feature reassembly operator that predicts location-specific kernels and applies them to local feature regions. Its design combines adaptive handling with efficient implementation and distinguishes it from related operators.

  • Formulation: CARAFE produces an output feature map of size C × σH × σW from an input map of size C × H × W.Each output location corresponds to a source location determined by the integer upsampling ratio σ.
  • Formulation: CARAFE predicts a reassembly kernel for each output location from the content of its corresponding input neighborhood.The kernel prediction module generates location-wise kernels, while the reassembly module applies them to the associated source region.
  • Kernel Prediction Module: The kernel prediction module uses channel compression, content encoding, and softmax normalization to generate efficient, spatially adaptive kernels.Channel compression reduces subsequent parameters and computation, while normalization makes each kernel sum to 1.
  • Content-aware Reassembly Module: The reassembly module computes each output feature through a weighted sum over a local k_up × k_up region.The weights determine how strongly individual pixels contribute to the target location.
  • Content-aware Reassembly Module: Unlike distance-based interpolation, CARAFE weights regional features according to content, allowing relevant local information to receive greater attention.The paper describes this content-dependent weighting as strengthening the semantics of the reassembled feature map.
  • Relation to Previous Operators: CARAFE differs from dynamic filtering by reassembling local features without learning cross-channel transformations, requiring only K × K kernel parameters per location.The paper contrasts this with dynamic filtering, whose per-location kernel has C × C × K × K parameters.

4. Applications of CARAFE

CARAFE can replace existing upsampling operators in dense prediction architectures without requiring structural changes. The paper applies it across detection, segmentation, and inpainting pipelines, including FPN, UperNet, and U-net-based models.

  • Applications: CARAFE integrates into existing dense prediction frameworks with negligible additional parameters.The paper presents applications in object detection, instance segmentation, semantic segmentation, and image inpainting.
  • Object Detection and Instance Segmentation: In FPN, CARAFE replaces nearest-neighbor interpolation in the top-down pathway to upsample low-resolution feature maps by 2x before fusion.The substitution is described as smooth and requiring no extra architectural change.
  • Object Detection and Instance Segmentation: CARAFE can also replace the deconvolution layer in the Mask R-CNN mask head, reducing computational cost while producing finer mask predictions.The layer upsamples predicted masks from 14 × 14 to 28 × 28.
  • Semantic Segmentation: In UperNet, CARAFE replaces original upsamplers in PPM, FPN, and FUSE, using a two-step strategy for the large PPM upsampling ratio.PPM first uses bilinear interpolation to reach half the original size, then CARAFE performs the remaining 2x upsampling.
  • Image Inpainting: In U-net-based inpainting models, CARAFE replaces the two upsampling layers, while mask propagation in Partial Conv is retained by updating masks with the reassembly kernels.The paper specifically evaluates this replacement in Global&Local and describes compatibility with Partial Conv.

5. Experiments

Experiments across detection, segmentation, and inpainting show that CARAFE improves performance while remaining efficient. Ablations further identify practical settings for balancing accuracy and computational cost.

  • Object Detection & Instance Segmentation: CARAFE improves Faster RCNN by 1.2% bbox AP and Mask RCNN by 1.3% mask AP on MS COCO.Improvements exceed 1% AP across small, medium, and large object scales.
  • Object Detection & Instance Segmentation: CARAFE achieves the best AP among tested Faster RCNN upsampling operators with relatively small FLOPs and parameter counts.Nearest + Conv and Bilinear + Conv add parameters without significant gains, while several learning-based alternatives perform worse.
  • Object Detection & Instance Segmentation: CARAFE achieves the best instance segmentation performance among tested operators when replacing the Mask R-CNN mask-head deconvolution layer.This comparison changes only the mask prediction branch and reports mask AP.
  • Semantic Segmentation: CARAFE raises UperNet single-scale mIoU on ADE20k val from 40.44% to 42.23%.It also outperforms recent strong baselines such as PSPNet and PSANet.
  • Ablation Study & Further Analysis: A channel-compression setting of Cm = 64 preserves performance while improving efficiency, whereas smaller values slightly reduce performance.Removing the channel compressor achieves the same performance, supporting its role in accelerating kernel prediction.
  • Ablation Study & Further Analysis: Increasing kencoder and kup together improves performance, with kencoder = kup − 2 a robust empirical choice.The default setting is kup = 5 and kencoder = 3 as a performance-efficiency trade-off.
  • Ablation Study & Further Analysis: Softmax and sigmoid normalization achieve the same performance and outperform unnormalized sigmoid, showing that kernel weights should sum to 1.The comparison evaluates alternative kernel-normalizer designs.

6. Conclusion

CARAFE is presented as a universal, lightweight upsampling operator that consistently improves dense prediction performance while adding little computational overhead.

  • CARAFE consistently improves object detection, instance segmentation, semantic segmentation, and inpainting performance on standard benchmarks.The reported gains are 1.2% AP, 1.3% AP, 1.8% mIoU, and 1.1 dB, respectively.
  • CARAFE introduces little computational overhead and can be readily integrated into modern network architectures.
  • The paper identifies low-level vision tasks such as image restoration and super-resolution as future application areas for CARAFE.

Appendix A. Detail Experimental Settings

The experiments use task-specific training and preprocessing configurations for object detection, instance segmentation, and image inpainting.

  • Object detection and instance segmentation use Faster R-CNN and Mask R-CNN with a ResNet-50 backbone and FPN.Inputs are resized while preserving aspect ratio, with an 800-pixel shorter edge or 1333-pixel longer edge.
  • The detection and segmentation models are trained with synchronized SGD using an initial learning rate of 0.02, momentum 0.9, and weight decay 0.0001.Training uses batch size 16 over 8 GPUs.
  • The inpainting experiments use free-form masks and replace convolution layers with the official Partial Convolution module for the Partial Conv model.
  • Inpainting models are trained with Adam at learning rate 0.0001, β1 = 0.5, β2 = 0.9, and batch size 32.Inputs and outputs are linearly scaled to [−1, 1].

Appendix B. Visualization of CARAFE

The visualization illustrates CARAFE’s content-aware reassembly in the top-down pathway of an FPN.

  • CARAFE reassembles red feature units into a green center unit during FPN top-down processing.

Appendix C. Visual Results Comparison

The visual comparisons show CARAFE against baseline models across object detection, instance segmentation, semantic segmentation, and image inpainting.

  • Figure 7 compares Mask R-CNN baseline results with Mask R-CNN using CARAFE on COCO 2017 validation data.
  • Figure 8 presents input images, ground truth, UperNet baseline results, and UperNet with CARAFE on ADE20K validation data.
  • Figure 9 compares masked inputs, Global&Local baseline results, Global&Local with CARAFE, and original images on Places validation data.
  • Figure 6 visualizes CARAFE’s content-aware reassembly in an FPN top-down pathway.
Loading 1905.02188v3…