PaperlayerView exact revision on arXiv

Source-linked AI summary

Frequency-aware Feature Fusion for Dense Image Prediction

Linwei Chen, Ying Fu, Lin Gu, Chenggang Yan, Tatsuya Harada, Gao Huang

arXiv:2408.12879v1cs.CVcs.AI

TL;DR

Dense prediction needs semantically informative features with accurate boundaries, but standard fusion can introduce intra-category inconsistency and boundary displacement. FreqFusion combines adaptive low-pass filtering, offset resampling, and high-pass enhancement to address these issues. It improves feature similarity and performance across semantic, object, instance, and panoptic segmentation tasks.

  • Problem

    Standard feature fusion can worsen intra-category inconsistency and boundary displacement when combining downsampled coarse features with high-resolution features.

  • Method

    FreqFusion uses adaptive low-pass filters, offset-based resampling, and adaptive high-pass filters to smooth inconsistent high-level features, replace them with consistent neighbors, and enhance lower-level boundary details.

  • Results

    FreqFusion consistently improves feature similarity and dense prediction performance across semantic segmentation, object detection, instance segmentation, and panoptic segmentation.

  • Takeaways & Limitations

    The method provides simultaneous feature consistency and boundary sharpness while integrating with existing dense prediction architectures.

  • Takeaways & Limitations

    Future work must investigate computational efficiency and real-time applicability in resource-constrained environments and extend the method to temporal inputs.

Abstract

from arXiv · show

Dense image prediction tasks demand features with strong category information and precise spatial boundary details at high resolution. To achieve this, modern hierarchical models often utilize feature fusion, directly adding upsampled coarse features from deep layers and high-resolution features from lower levels. In this paper, we observe rapid variations in fused feature values within objects, resulting in intra-category inconsistency due to disturbed high-frequency features. Additionally, blurred boundaries in fused features lack accurate high frequency, leading to boundary displacement. Building upon these observations, we propose Frequency-Aware Feature Fusion (FreqFusion), integrating an Adaptive Low-Pass Filter (ALPF) generator, an offset generator, and an Adaptive High-Pass Filter (AHPF) generator. The ALPF generator predicts spatially-variant low-pass filters to attenuate high-frequency components within objects, reducing intra-class inconsistency during upsampling. The offset generator refines large inconsistent features and thin boundaries by replacing inconsistent features with more consistent ones through resampling, while the AHPF generator enhances high-frequency detailed boundary information lost during downsampling. Comprehensive visualization and quantitative analysis demonstrate that FreqFusion effectively improves feature consistency and sharpens object boundaries. Extensive experiments across various dense prediction tasks confirm its effectiveness. The code is made publicly available at https://github.com/Linwei-Chen/FreqFusion.

1 INTRODUCTION

Dense prediction requires both category-discriminative features and precise boundaries, but standard fusion can create intra-category inconsistency and boundary displacement. FreqFusion addresses these issues with frequency-aware filtering and resampling, improving feature similarity and performance across dense prediction tasks.

  • Motivation: Hierarchical models fuse coarse high-level features with high-resolution lower-level features because dense prediction requires category information and spatial boundary detail.Downsampling loses boundary information, while higher-level and lower-level features provide complementary semantic and spatial information.
  • Problems with standard fusion: Standard fusion can amplify intra-category inconsistency by replicating inconsistent features during interpolation, while excessive smoothing can displace boundaries.Feature similarity analysis measures intra-category similarity, inter-class similarity, and their margin to quantify these problems.
  • Experiments: FreqFusion improves semantic segmentation, object detection, instance segmentation, and panoptic segmentation results across representative baselines.Reported gains include 2.8 and 2.0 mIoU for SegFormer-B1 and SegNeXt-T, 1.8 AP for Faster R-CNN-R50, 1.7 box AP and 1.3 mask AP for Mask R-CNN-R50, and 1.9 PQ for Panoptic FPN-R50.
  • Proposed method: FreqFusion combines spatially variant low-pass filtering, offset-based resampling, and high-pass enhancement to improve category consistency and boundary sharpness.The ALPF attenuates disturbed high frequencies within objects, the offset generator replaces inconsistent features with nearby consistent ones, and the AHPF restores lower-level high-frequency details.
  • Evidence: FreqFusion increases intra-category similarity and similarity margin while producing more consistent features and clearer boundaries.Visualizations and quantitative analyses report improved feature consistency and sharper predictions.

2 RELATED WORK

Related work covers dense prediction architectures, feature fusion and aggregation, feature upsampling and sampling, and frequency-domain learning. FreqFusion differs by quantitatively defining fusion problems and jointly targeting feature consistency and boundary sharpness.

  • Dense Image Prediction: Dense prediction architectures use feature fusion because downsampling reduces spatial detail needed for accurate classification and localization.FreqFusion is designed for integration with CNN and Transformer architectures.
  • Feature Fusion and Aggregation: Feature fusion combines lower-level high-resolution features with higher-level coarse features, while feature aggregation chains fusion operations across network stages.Top-down and bottom-up aggregation architectures address resolution and semantic-level gaps.
  • Improved Feature Fusion: Existing fusion improvements include kernel-based methods that learn data-dependent upsampling kernels and sampling-based methods that adjust feature sampling coordinates.Examples of sampling-based approaches use predicted offsets or deformable convolution for alignment.
  • Relationship to Prior Work: FreqFusion extends prior fusion methods by extracting high-frequency information from low-level features and incorporating it residually.It also defines intra-category inconsistency and boundary displacement through feature similarity analysis rather than relying only on empirical observations.
  • Frequency-Domain Learning: Frequency-domain learning has been used to study optimization, generalization, spectral bias, and the effects of frequency perturbations in deep networks.Prior work links disturbed high frequencies with reduced intra-category similarity, motivating adaptive low-pass and high-pass filtering in FreqFusion.

3 FEATURE SIMILARITY ANALYSIS METRICS

The paper introduces feature-similarity metrics to quantify intra-category inconsistency and boundary displacement in fused features. Intra-category similarity, similarity margin, and similarity accuracy collectively assess feature discriminative power and category separation.

  • Feature-similarity analysis quantifies intra-category inconsistency and boundary displacement during feature fusion.These metrics establish a basis for developing and analyzing feature-fusion techniques.
  • Similarity margin equals intra-category similarity minus inter-category similarity, measuring the separation between a feature’s own and other categories.A larger gap between intra- and inter-category similarities is associated with reduced misclassification risk.
  • Intra-category similarity compares each feature vector with its category center using cosine similarity.For binary segmentation, category centers are derived separately for the relevant ground-truth categories.
  • Similarity accuracy assigns each feature to its most similar category center and measures cases where inter-category similarity exceeds intra-category similarity.It therefore assesses misclassification risk caused by intra-category inconsistency and boundary displacement.
  • Together, the three metrics evaluate category information, discriminative power, and separation between different categories in feature maps.

4 FREQUENCY-AWARE FEATURE FUSION

FreqFusion replaces straightforward feature fusion with frequency-aware processing that smooths inconsistent object features, resamples toward consistent neighbors, and restores boundary detail. Its visualizations and quantitative analyses show improved feature consistency and sharper, more accurate boundaries.

  • Design of FreqFusion: FreqFusion combines an Adaptive Low-Pass Filter generator, offset generator, and Adaptive High-Pass Filter generator.The framework uses initial and final fusion stages to provide inputs for these three generators.
  • Standard feature fusion: Standard fusion can amplify intra-category inconsistency and boundary displacement by upsampling inconsistent features and blurring boundary information.It commonly upsamples coarse features through nearest-neighbor or bilinear interpolation before addition or concatenation.
  • Adaptive filtering and resampling: The ALPF generator predicts spatially variant low-pass filters that attenuate within-object high frequencies, while the offset generator resamples toward more consistent neighboring features.The offset generator is intended to address large inconsistent regions and refine thin or boundary areas that smoothing alone may not correct.
  • Quantitative analysis: Overall intra-category similarity rises from 0.727 to 0.799, similarity margin from 0.245 to 0.297, and similarity accuracy from 0.918 to 0.941 with the ALPF generator.These Table 1 results support improved feature consistency within the FreqFusion framework.
  • Offset generator: The offset generator improves feature consistency and boundary delineation by directing samples toward clearer interior or boundary locations.Visualizations show improved consistency and more accurate boundaries after offset-based resampling.
  • Adaptive High-Pass Filter Generator: The AHPF generator enhances detailed boundary information, improving the clarity of bus outlines and a person’s head details.It predicts spatially variant high-pass filters from the initially fused feature and applies them residually.

5 EXPERIMENTAL RESULTS

The experiments evaluate FreqFusion across four dense prediction tasks using established model and upsampling configurations.

  • FreqFusion is evaluated on semantic segmentation, object detection, instance segmentation, and panoptic segmentation.The study examines its universality across four representative dense prediction tasks.
  • The experiments set the Deconvolution and Pixel Shuffle kernel sizes to 3.
  • CARAFE uses its default configuration, while IndexNet and A2U use specified versions for experimental stability.

5.1 Semantic Segmentation

FreqFusion targets semantic segmentation by improving feature consistency within objects and sharpening boundaries during feature fusion. Across model structures and datasets, it improves segmentation accuracy while maintaining a minor efficiency impact.

  • Motivation: Semantic segmentation requires feature fusion that supports consistent pixel grouping and precise object boundaries.The section evaluates feature fusion using mIoU for overall segmentation and bIoU for boundary delineation.
  • Quantitative results: FreqFusion improves SegFormer-B1 by 2.8 mIoU on ADE20K, exceeding the second-place Dysample-S+ by 1.2 mIoU.Table 2 compares mIoU and bIoU on the ADE20K validation set.
  • Qualitative results: Visualizations report more consistent fused features, sharper boundaries, and improved segmentation accuracy and consistency on Cityscapes and ADE20K.The comparisons use standard feature fusion and vanilla SegNeXt or SegFormer baselines.
  • Model generality: FreqFusion consistently improves diverse architectures, including CNN-, Transformer-, FPN-, and concatenation-based models, with minor extra parameters and computation.It is also applied to large MaskFormer models, increasing mIoU from 53.9 to 55.3 with Swin-B and from 56.1 to 56.8 with Swin-L.
  • Cross-dataset results: FreqFusion improves SegNeXt by 1.0, 2.4, and 2.0 on Cityscapes, ADE20K, and COCO-stuff, respectively.The gains are reported across multiple challenging datasets.
  • Efficiency: FreqFusion improves SegNeXt by 2.4 mIoU while reaching 23.0 FPS, compared with Dysample's 1.1 mIoU gain and 25.9 FPS.The method is slightly slower but achieves the reported higher accuracy improvement.

5.2 Object Detection

FreqFusion is evaluated as a replacement for FPN feature fusion in Faster R-CNN on MS COCO. It improves object detection performance with both ResNet-50 and ResNet-101 backbones.

  • Motivation and evaluation: Object detection requires feature fusion that supports accurate localization and precise object classification.The experiments use COCO Average Precision metrics, including AP, AP50, AP75, APS, APM, and APL.
  • Results: FreqFusion improves COCO detection performance by 1.9 AP, surpassing all listed competing methods.The comparison uses Faster R-CNN with FPN and reports results in Table 9 and Figure 15.
  • Results: FreqFusion leads Dysample+ by 0.7 AP and achieves comparable 39.4 AP with ResNet-50 relative to the more robust ResNet-101.The reported comparison emphasizes both margin over the runner-up and backbone robustness.
  • Results: With ResNet-101, FreqFusion delivers a 1.6 AP improvement and outperforms Dysample+ by 0.5 AP.The modification is confined to feature fusion stages within the FPN.

5.3 Instance Segmentation

FreqFusion is evaluated in Mask R-CNN instance segmentation on MS COCO, where feature fusion must preserve category consistency and object boundaries. It improves both mask and box detection metrics.

  • Motivation and evaluation: Instance segmentation requires consistent category information and precise delineation of individual object boundaries.Evaluation uses standard Box AP and Mask AP on MS COCO.
  • Results: FreqFusion improves COCO performance by 1.3 mask AP and 1.7 box AP, achieving the leading result over listed competitors.The experiments use Mask R-CNN with ResNet backbones and modify FPN feature fusion stages.
  • Results: FreqFusion exceeds Dysample+ by 0.3 mask AP and 0.4 box AP, while ResNet-50 achieves comparable results at 36.0 mask AP and 40.0 box AP.With ResNet-101, the reported improvements are 1.4 mask AP and 1.6 box AP.

5.4 Panoptic Segmentation

FreqFusion is evaluated for panoptic segmentation on COCO using Panoptic FPN, where it improves PQ over competing feature-fusion methods and remains competitive across backbones.

  • The study uses MS COCO with 80 object categories, reporting PQ, SQ, and RQ, and modifies only the FPN upsampling stages of Panoptic FPN.Panoptic FPN uses ResNet-50 and three FreqFusion modules for 4×, 8×, 16×, and 32× downsampled features.
  • FreqFusion improves panoptic segmentation performance by 2.5 PQ on COCO and surpasses CARAFE, IndexNet, A2U, FADE, SAPA-B, Dysample-S+, and Dysample+.The evaluation uses ResNet as the backbone and compares against recent state-of-the-art feature fusion methods.
  • FreqFusion leads Dysample+ by 1.2 PQ on the COCO panoptic segmentation evaluation.
  • With ResNet-50, FreqFusion achieves 42.7 PQ, exceeding the 42.2 PQ obtained with the more robust ResNet-101.
  • With ResNet-101, FreqFusion delivers a 1.8 PQ improvement and a 1.0 AP lead over Dysample+.

5.5 Ablation Studies

Ablation studies on ADE20K examine FreqFusion’s generators and initial fusion, showing that adaptive filtering improves segmentation performance and that combining low- and high-pass processing is especially effective.

  • Using only the ALPF generator improves mIoU by +0.3 in the enhanced initial-fusion ablation.
  • Combining ALPF and AHPF generators raises performance to 43.5 in the enhanced initial-fusion ablation.The result highlights the contribution of both generators to refining segmentation features during initial fusion.
  • The initial-fusion ablation addresses limitations of bilinear interpolation, including blurred boundaries, on the ADE20K validation set.

6 DISCUSSION WITH RELATED WORKS

FreqFusion differs from prior kernel- and sampling-based feature-fusion methods by explicitly addressing feature inconsistency and boundary displacement through frequency-aware smoothing, resampling, and high-frequency enhancement.

  • A2U and IndexNet rely exclusively on low-level features for dynamic kernels, while CARAFE relies solely on high-level features and overlooks high-resolution structure.
  • FreqFusion smooths high-level features, uses local similarity as guidance, and resamples features to replace inconsistent representations.
  • Unlike prior methods that mainly improve upsampling, FreqFusion extracts high-frequency information from low-level features and adds it residually to enhance feature fusion.
  • FreqFusion defines intra-category inconsistency and boundary displacement quantitatively through feature similarity analysis, unlike prior empirical treatments.

7 CONCLUSION

The paper introduces FreqFusion to address intra-category inconsistency and boundary displacement in dense prediction, using adaptive filtering and offset-based resampling, while identifying computational efficiency and temporal inputs as future directions.

  • FreqFusion targets intra-category inconsistency and boundary displacement in dense image prediction tasks.
  • FreqFusion combines ALPF, Offset, and AHPF generators to adaptively smooth high-level features, resample nearby category-consistent features, and enhance high-frequency details.
  • Future work should investigate computational efficiency and real-time applicability in resource-constrained environments before practical deployment.
  • Extending FreqFusion to temporal inputs such as videos remains a future direction for handling motion blur or occlusion in visual perception tasks.
Loading 2408.12879v1…