PaperlayerView exact revision on arXiv

Source-linked AI summary

Enabling Memory-efficient Im2win Convolution with Multi-precision Support on GPU CUDA and Tensor Cores

Xiang Fu, Jixiang Ma, Xinpeng Zhang, Peng Zhao, Shuai Lu, Xu Tony Liu

arXiv:2608.20725v1cs.DCcs.CV

TL;DR

Convolution dominates CNN execution, but GPU implementations face memory, locality, precision, and kernel-size trade-offs. The paper extends im2win with full-precision CUDA-core and half-precision tensor-core kernels plus hardware-oriented optimizations. Across twelve CNN benchmarks, im2win outperforms the cited cuDNN and cuBLAS baselines while using less memory.

  • Problem

    GPU convolution methods face large memory overhead, poor cache utilization, limited kernel-size effectiveness, or numerical instability, motivating a more efficient multi-precision approach.

  • Method

    The paper extends im2win with full-precision CUDA-core kernels and half-precision tensor-core kernels, adding index precomputation, zig-zag access, and asynchronous data movement.

  • Results

    Across twelve CNN benchmarks, Im2win_TC achieves 1.4× higher overall TFLOPS than cuDNN_TC and 6.4× higher than im2col_cuBLAS_TC.

  • Takeaways & Limitations

    The results support im2win as a unified convolution framework spanning CUDA cores and tensor cores with optimized multi-precision execution.

Abstract

from arXiv · show

Convolution is a principal computational bottleneck in deep neural networks, and its efficiency depends on tight integration between algorithms and GPU hardware. Existing GPU convolution methods suffer from large memory overhead, poor cache utilization, limited effectiveness across kernel sizes, or numerical instability. This work extends the im2win paradigm -- a universal, memory-efficient convolution method with contiguous memory access for all kernel sizes -- to run efficiently in full precision on CUDA cores and half precision on tensor cores. By introducing new kernel designs and optimizations such as zig-zag memory access and asynchronous data movement, im2win efficiently exploits hardware-accelerated half-precision matrix multiply-accumulate operations. Across twelve CNN benchmarks, im2win achieves up to 2.8x higher TFLOPS than its CUDA core implementation, 1.4x higher than cuDNN, and 6.4x higher than GEMM-based convolution with cuBLAS, while using as little as 53% and 35% of their memory, respectively. These results establish im2win as a unified, high-performance convolution framework for modern GPU architectures.

I. Introduction

GPU convolution is a major CNN bottleneck, while existing approaches trade memory, locality, kernel-size performance, or numerical stability. This work extends im2win with multi-precision GPU kernels and evaluates optimized implementations against cuDNN and cuBLAS-based convolution.

  • Motivation: 50–90% of execution time is attributed to convolution operations in CNN workloads.Efficient GPU convolution must use CUDA cores for full precision and tensor cores for mixed-precision acceleration.
  • Motivation: Existing methods expose distinct trade-offs: direct convolution has poor cache utilization, FFT is slow for small filters, Winograd can be numerically unstable, and im2col incurs high memory overhead.Im2col-based GEMM methods can also produce irregular matrices and suboptimal performance.
  • Im2win approach: Im2win transforms the input into a compact tensor with contiguous access and data reuse, reducing memory consumption relative to im2col.The paradigm targets both the memory overhead of im2col and the poor locality of direct convolution.
  • Contributions: The GPU implementation supports full-precision kernels on CUDA cores and half-precision kernels on tensor cores.The design adds index precomputation, zig-zag memory access, and asynchronous data movement for tensor-core utilization.
  • Evaluation: The evaluation compares optimized im2win with cuDNN variants and PyTorch’s im2col-based cuBLAS convolution across multiple precisions and GPU core types.The study uses twelve CNN benchmarks and includes an ablation of individual optimizations.

III. Im2win

Im2win represents convolution windows in a compact arrangement that preserves contiguous access while avoiding much of im2col’s redundancy. Its memory advantage depends on overlap between neighboring windows.

  • A. Notation: The convolution maps an input tensor and filter tensor to an output tensor in NCHW layout.The notation identifies batch, channel, height, and width dimensions for input, filter, and output tensors.
  • A. Notation: NCHW stores width elements contiguously, making width access cheapest and influencing GPU data-layout choices.Access costs increase across height, channel, and batch dimensions in the described layout.
  • B. Im2win Data Transformation: Im2win arranges elements from each sliding window consecutively across channels.The transformation is illustrated using successive red, blue, and green windows for successive output elements.
  • B. Im2win Data Transformation: Im2col duplicates overlapping window elements across rows, producing substantially greater memory usage than the original input.Im2win avoids this redundant storage through a different tensor arrangement.
  • B. Im2win Data Transformation: When stride s is less than filter height H_f, im2win requires less memory than im2col; when s = H_f, their space usage matches.The stated savings are N_o×C_o×H_o×W_o×C_i×H_f×(W_f−s).

C. Im2win Convolution

Naive im2win computes convolution through nested loops over output and filter dimensions, while preserving a compact representation that supports reuse across overlapping windows.

  • C. Im2win Convolution: For each output element, im2win multiplies the corresponding window and filter elements and sums the products.Successive output elements use successive sliding windows shifted according to the stride.
  • C. Im2win Convolution: Im2win has time complexity O(N_o×C_o×H_o×W_o×C_f×H_f×W_f), matching direct convolution.The computation uses the same asymptotic operation count as direct convolution.
  • C. Im2win Convolution: Filter layout and cache reuse across overlapping windows reduce memory loading time and improve efficiency relative to direct and im2col convolution.The filter layout prioritizes height before width to align with the im2win access pattern.
  • C. Im2win Convolution: The tensor-core algorithm organizes computation around output tiles with M = C_o, N = N_o×H_o×W_o, and K = C_f×H_f×W_f.The implementation maps these dimensions onto thread blocks and WMMA registers.

D. Im2win Convolution on Tensor Cores

The tensor-core implementation tiles im2win outputs and maps them to WMMA computations, using layouts and buffering strategies intended to improve memory access and hide movement latency.

  • D. Im2win Convolution on Tensor Cores: The tensor-core implementation uses NHWC layout to improve memory-access contiguity for convolutional windows.The design also considers thread-block organization, shared memory, and warp-level computation.
  • D. Im2win Convolution on Tensor Cores: The algorithm tiles the output tensor and processes all batches simultaneously because im2win requires little extra memory.Output indices are mapped to M = C_o and N = N_o×H_o×W_o.
  • D. Im2win Convolution on Tensor Cores: WMMA’s 16×16 granularity constrains block dimensions to multiples of 16, with adopted block shapes of 32×128 and 64×128.Larger blocks improve tensor-core utilization but increase shared-memory pressure and reduce concurrent blocks.
  • D. Im2win Convolution on Tensor Cores: Vectorized loads move convolution data through global memory, shared memory, and registers, while asynchronous movement and double buffers hide latency.Two shared-memory buffers and two registers support the overlapping movement strategy.

E. Optimizations for im2win convolutions on CUDA and Tensor Cores

The optimization suite combines standard GPU techniques with methods tailored to NVIDIA SM 80 tensor-core execution. Its goal is to maximize computational throughput and memory bandwidth.

  • The design uses tiling, data pre-fetching, shared-memory and register management, vectorized loads and stores, and double buffering.
  • It additionally introduces asynchronous data movement and index precomputation for NVIDIA SM 80 architecture.

1) index precomputation:

The implementation reduces convolution overhead through precomputed memory-access offsets, asynchronous movement, zig-zag access, and precision-specific GPU kernels. These choices target redundant computation, latency, bank conflicts, and hardware utilization.

  • 1) index precomputation:: Precomputing fixed thread index offsets on the host and storing them in constant memory avoids redundant device-side index calculations.
  • Asynchronous data movement overlaps data transfer with computation to hide movement latency, using PTX instructions and double buffering.
  • Zig-zag access rearranges shared-memory access order to alleviate bank conflicts when thread-block division creates large dimensions.
  • The method provides an FP32 CUDA-core variant and an FP16 tensor-core variant, using WMMA to exploit native half-precision matrix multiply-accumulate.

A. Experimental Setup

The evaluation uses an RTX 3090 system and compares im2win with cuBLAS- and cuDNN-based convolutions across twelve diverse convolutional benchmarks. Results distinguish FP32 CUDA-core execution from FP16 tensor-core execution.

  • A. Experimental Setup: Experiments run on an NVIDIA GeForce RTX 3090 with 24 GB memory and Ampere architecture featuring third-generation tensor cores.
  • The software stack uses CUDA 11.3, PyTorch 2.2.0, cuBLAS 11.3, and cuDNN 8.2.1, while peak GPU memory is monitored with nvidia-smi.
  • The comparison includes optimized cuDNN approaches spanning im2col, implicit GEMM, FFT, and Winograd, with cuDNN selecting algorithms dynamically.
  • The study evaluates FP32 CUDA-core and FP16 tensor-core implementations of im2win and PyTorch’s cuBLAS-backed convolution.
  • B. Benchmarks: The benchmark suite contains twelve unique convolutional layers to cover varied kernel dimensions beyond uniform-kernel models.

C. Overall Performance on CUDA and Tensor Cores

Across the evaluated benchmarks, tensor-core im2win substantially outperforms its CUDA-core, cuDNN, and cuBLAS counterparts while using less memory than the baselines on average. The figure reports TFLOPS and memory usage for the three convolutional algorithms on both core types.

  • C. Overall Performance on CUDA and Tensor Cores: 2.7× higher overall TFLOPS is achieved by im2win_TC than im2win_CC.FP16 processes twice as many elements per data-transfer volume, while tensor cores execute block-level matrix operations.
  • 1.4× higher overall TFLOPS is achieved by im2win_TC than cuDNN_TC, and 6.4× higher than im2col_cuBLAS_TC.
  • The comparison covers TFLOPS and memory usage for im2win, im2col_cuBLAS, and cuDNN on CUDA and tensor cores.
  • 53% as much memory as cuDNN_TC and 35% as much as im2col_cuBLAS_TC are used on average by im2win_TC.im2win_TC uses less memory than cuDNN_TC except on cv3 and cv4.

D. Micro Benchmark with Each Convolution

The comparison evaluates im2win against enforced cuDNN variants and cuBLAS-based convolution on CUDA and tensor cores, measuring TFLOPS and memory usage across benchmarks.

  • The evaluation enforces specific cuDNN algorithms because algorithmic constraints prevent some variants from executing on particular layers.
  • Im2win achieves the highest TFLOPS on six CUDA-core benchmarks, while cuDNN_IPG leads on the other six.
  • Im2win minimizes CUDA-core memory usage in every benchmark except cv4, where it slightly exceeds cuBLAS_im2col.
  • Im2win delivers peak tensor-core TFLOPS on eight benchmarks; cuDNN_IPG leads on cv3 and cv4, while cuDNN_WN leads on cv11.

1) Im2win vs cuBLAS_im2col vs cuDNN_im2col:

Im2win improves convolution performance while reducing memory relative to cuBLAS im2col and several cuDNN baselines, using compact window transformations and full-batch processing.

  • 3.4× speedup over cuBLAS_im2col_CC and 2.7× speedup over cuDNN_im2col_CC are achieved by im2win_CC overall.
  • 6.4× speedup compared to cuBLAS_im2col_TC is achieved by im2win_TC.
  • Im2win_CC uses about 62% as much memory as cuBLAS_im2col_CC and about 41% as much as cuDNN_im2col_CC on average.
  • Im2win_TC averages 55% of cuBLAS_im2col_TC memory usage, while full-batch processing can increase memory relative to cuDNN’s mini-batch approach.Im2win_TC has the smallest memory requirement except on cv3, cv4, and cv8.
  • Im2win’s compact window transformation reduces redundant storage while improving element reuse and spatial locality.
  • Compared with cuDNN_WN, im2win reaches 2.5× higher overall TFLOPS on CUDA cores and 2.8× on tensor cores, using 27% and 37% of its memory, respectively.Winograd transformations reduce multiplications but introduce additional memory overhead.

4) Im2win vs cuDNN_FT:

Im2win is compared with FFT-based convolution and evaluated through tensor-core ablations of three optimizations, revealing performance trade-offs across transformations and kernel-window sizes.

  • 4) Im2win vs cuDNN_FT:: FFT outperforms im2win on cv4 because frequency-domain transformation overhead becomes more pronounced for larger tensor sizes.
  • Optimization ablation: Double buffering generally provides the greatest ablation-study gain, followed by asynchronous data movement, while Zig-Zag access provides the smallest improvement.
  • Optimization ablation: Removing double buffering improves performance for small convolution windows because synchronization latency and additional shared-memory use outweigh its benefits.
  • Optimization ablation: Zig-Zag access primarily reduces shared-memory bank conflicts, but its marginal benefit remains limited after prefetching from other optimizations.
  • Conclusion: The implementation supports full precision on CUDA cores and half precision on tensor cores, with double buffering identified as the most critical optimization.
Loading 2608.20725v1…