PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning in the Frequency Domain

Kai Xu, Minghai Qin, Fei Sun, Yuhao Wang, Yen-Kuang Chen, Fengbo Ren

arXiv:2002.12416v4cs.CV

TL;DR

Fixed-size spatial CNN inputs force large images to be downsampled, risking information loss and accuracy degradation. The paper instead learns from DCT-domain inputs and selects salient frequency channels, achieving higher accuracy than spatial downsampling while reducing input data size across several vision tasks.

  • Problem

    Fixed-size CNN inputs require large images to be downsampled, which removes information and can degrade accuracy.

  • Method

    The method transforms images into DCT coefficients, feeds them to minimally modified CNNs, and learns which frequency channels can be removed.

  • Results

    Across classification, detection, and segmentation, frequency-domain learning improves accuracy over spatial downsampling; ResNet-50 can prune up to 87.5% of frequency channels with no or little degradation.

  • Takeaways & Limitations

    Static selection of salient frequency channels can preserve or improve task accuracy while reducing input data size and communication requirements.

Abstract

from arXiv · show

Deep neural networks have achieved remarkable success in computer vision tasks. Existing neural networks mainly operate in the spatial domain with fixed input sizes. For practical applications, images are usually large and have to be downsampled to the predetermined input size of neural networks. Even though the downsampling operations reduce computation and the required communication bandwidth, it removes both redundant and salient information obliviously, which results in accuracy degradation. Inspired by digital signal processing theories, we analyze the spectral bias from the frequency perspective and propose a learning-based frequency selection method to identify the trivial frequency components which can be removed without accuracy loss. The proposed method of learning in the frequency domain leverages identical structures of the well-known neural networks, such as ResNet-50, MobileNetV2, and Mask R-CNN, while accepting the frequency-domain information as the input. Experiment results show that learning in the frequency domain with static channel selection can achieve higher accuracy than the conventional spatial downsampling approach and meanwhile further reduce the input data size. Specifically for ImageNet classification with the same input size, the proposed method achieves 1.41% and 0.66% top-1 accuracy improvements on ResNet-50 and MobileNetV2, respectively. Even with half input size, the proposed method still improves the top-1 accuracy on ResNet-50 by 1%. In addition, we observe a 0.8% average precision improvement on Mask R-CNN for instance segmentation on the COCO dataset.

1. Introduction

The paper replaces spatial resizing with frequency-domain processing to preserve information while reducing input data, and uses learned channel selection to remove trivial frequencies. Across classification and instance segmentation, this approach improves accuracy and supports substantial frequency-channel pruning.

  • Motivation: Large images are commonly downsized to fixed CNN input resolutions, but this removes information and can degrade accuracy.Prior task-aware downsizing methods require additional computation and are task-specific.
  • Method: The proposed method transforms high-resolution images into DCT coefficients and feeds the reshaped frequency-domain representation to existing CNN architectures.The method requires little modification to CNNs that normally accept RGB input.
  • Frequency sensitivity: CNNs are more sensitive to low-frequency channels than high-frequency channels, consistent with the reported human visual system comparison.The observation motivates channel selection based on frequency importance.
  • Results: +1.60% on ResNet-50, +0.63% on MobileNetV2, and +0.8% on Mask R-CNN are reported against the conventional spatial-downsampling approach.The gains cover ImageNet classification, object detection, and instance segmentation tasks.
  • Channel selection: 87.5% of frequency channels can be pruned on ResNet-50 with no or little ImageNet accuracy degradation.Static selection preserves salient channels while reducing input data size and transmission requirements.

2. Related Work

Prior work studies compressed frequency representations, frequency-domain features, model conversion, and efficient network computation. The paper distinguishes its approach by operating on raw inputs and selecting salient frequency components to reduce communication bandwidth.

  • Learning in the frequency domain: Prior methods jointly train autoencoder-based networks for compression and inference, extract frequency-domain features for classification, or convert spatial CNNs to the frequency domain.These approaches represent several existing directions for learning in the frequency domain.
  • Efficient network training: Other efficient-network methods adapt computation to input-dependent intermediate features, whereas this method operates on raw inputs.The paper frames raw-input frequency selection as a way to reduce input communication bandwidth.

3. Methodology

The paper replaces spatial preprocessing with frequency-domain inputs and learns which DCT channels matter for inference. It then uses those channel-importance patterns to support static pruning while retaining accuracy and reducing input data.

  • Data Pre-processing in the Frequency Domain: High-resolution RGB images are transformed to YCbCr and the frequency domain before inference, grouping equal-frequency DCT coefficients into channels.The preprocessing pipeline includes resizing, cropping, flipping, DCT transformation, and DCT reshaping.
  • Data Pre-processing in the Frequency Domain: The frequency-domain representation can connect to existing CNNs with minimal modification by removing the initial input layers and adjusting the next layer’s channel count.For ResNet-50, the frequency input is connected to the first residual block after adapting the input channels.
  • Learning-based Frequency Channel Selection: The method balances task accuracy against the number of selected channels through a regularized loss, enabling trivial frequency components to be pruned for inference.Static selection uses the learned channel importance to reduce input data size, computational complexity, and communication bandwidth while maintaining inference accuracy.
  • Learning-based Frequency Channel Selection: A learning-based gate assigns binary on/off decisions to frequency channels, using SE-Block-derived features and trainable probabilities for channel selection.The gate produces a 1 × 1 × C × 2 tensor whose normalized values represent the probability that each channel is on or off.
  • Static Frequency Channel Selection: Low-frequency and luma Y channels are selected more often than high-frequency and chroma Cb/Cr channels across classification and segmentation.The shared heat-map pattern suggests these selection tendencies are not specific to one task, although some slightly higher-frequency channels are preferred over lower-frequency alternatives.

4. Experiment Results

Experiments evaluate frequency-domain learning across classification, detection, and segmentation, showing that selected frequency channels can preserve or improve accuracy while reducing input size.

  • The study benchmarks frequency-domain learning on image classification, detection, and segmentation tasks.
  • Image Classification: 1.4% higher top-1 accuracy is achieved by ResNet-50 frequency-domain models using selected channels and half the baseline input data size.The benefit is largely insensitive to the precise selected-channel shape among DCT-24D, DCT-24T, and DCT-24S.
  • Image Classification: 1.4% higher top-1 accuracy is achieved over the baseline ResNet-50 when all 192 frequency channels are used.The spatial YCbCr transformation lowers accuracy by roughly 0.5% relative to RGB, while the frequency-domain method performs better than the YCbCr case.
  • Image Classification: 0.454% and 0.074% top-1 accuracy improvements are obtained with 12 and 6 frequency channels on MobileNetV2, respectively.Selecting 32 and 24 channels improves top-1 accuracy by 0.662% and 0.580%, respectively.
  • Instance Segmentation: 0.4 bbox AP and 0.4 mask AP improvements are obtained with DCT-24S using half the RGB-based Mask R-CNN input data size.The comparison is against the RGB-based Mask R-CNN baseline.
  • Instance Segmentation: Frequency-domain Mask R-CNN produces the reported COCO instance-segmentation results, with qualitative examples shown in Figure 6.

5. Conclusion

The paper concludes that learning in the frequency domain generalizes across classification, detection, and segmentation while requiring little modification to RGB-input CNNs. Dynamic selection can prune most frequency channels with no or little accuracy degradation.

  • The method is demonstrated across classification, detection, and segmentation tasks and is designed to apply broadly to existing CNN training and inference methods.
  • Up to 87.5% of frequency channels can be pruned with no or little accuracy degradation across the evaluated tasks.

Supplementary Material for Learning in the Frequency Domain

The supplementary document provides further quantitative and qualitative insights into the reported results.

  • The supplement provides further quantitative and qualitative insights into the results.

A. Instructions to Reproduce the Experiments

The authors provide source code and instructions for reproducing the classification and segmentation experiments.

  • Source code based on PyTorch is provided for reproducing the image-classification and segmentation experiments.The repository contains separate classification and segmentation folders with pretrained models and instructions.

B. Additional Instance Segmentation Results

Figure 7 presents additional instance segmentation examples, extending the paper’s visual evidence for this task.

  • Figure 7 shows more instance segmentation examples.
  • The figure provides additional visual results for instance segmentation.
  • These examples supplement the instance segmentation results presented elsewhere in the paper.

C. Object Detection Results on Faster R-CNN

The paper evaluates a frequency-domain Faster R-CNN for object detection on COCO, using a frequency-domain ResNet-50 and feature pyramid network backbone. The proposed method achieves a 0.8% AP improvement over baseline Faster R-CNN.

  • Experimental setup: The object detection model uses frequency-domain ResNet-50 with a feature pyramid network as its backbone.The ResNet-50 model is fine-tuned with classification and bounding box regression heads.
  • Experimental setup: Evaluation uses the COCO train2017 split for training and val2017 split for evaluation.
  • Results: 0.8% AP improvement is achieved by frequency-domain Faster R-CNN compared with baseline Faster R-CNN on COCO.
Loading 2002.12416v4…