PaperlayerView exact revision on arXiv

Source-linked AI summary

TResNet: High Performance GPU-Dedicated Architecture

Tal Ridnik, Hussam Lawen, Asaf Noy, Emanuel Ben Baruch, Gilad Sharir, Itamar Friedman

arXiv:2003.13630v3cs.CVcs.LGeess.IV

TL;DR

Recent models may achieve strong ImageNet accuracy with fewer FLOPs yet run slower on GPUs than ResNet50, exposing a gap between FLOPs and practical throughput. The paper introduces GPU-dedicated TResNet architectures and implementation refinements, achieving improved speed-accuracy trade-offs across vision tasks, including 80.8% ImageNet top-1 accuracy at ResNet50-like throughput.

  • Problem

    FLOPs do not reliably predict GPU training and inference throughput, while GPU training speed and maximal batch size are often overlooked in network design.

  • Method

    The paper combines ResNet-based architectural refinements with code optimizations targeting GPU utilization, including SpaceToDepth, anti-alias downsampling, Inplace-ABN, block redesign, and optimized SE layers.

  • Results

    TResNet improves the speed-accuracy trade-off across tested datasets, including 80.8% ImageNet top-1 accuracy at GPU throughput similar to ResNet50’s 79.0%, plus strong downstream task performance.

  • Takeaways & Limitations

    TResNet provides GPU-dedicated models that outperform existing models along the ImageNet top-1 accuracy curve and generalize to classification, multi-label, and detection tasks.

  • Takeaways & Limitations

    The evaluation highlights GPU inference and training speed and maximal batch size, while inference-tailored deployment optimizations were intentionally omitted for fair comparison.

Abstract

from arXiv · show

Many deep learning models, developed in recent years, reach higher ImageNet accuracy than ResNet50, with fewer or comparable FLOPS count. While FLOPs are often seen as a proxy for network efficiency, when measuring actual GPU training and inference throughput, vanilla ResNet50 is usually significantly faster than its recent competitors, offering better throughput-accuracy trade-off. In this work, we introduce a series of architecture modifications that aim to boost neural networks' accuracy, while retaining their GPU training and inference efficiency. We first demonstrate and discuss the bottlenecks induced by FLOPs-optimizations. We then suggest alternative designs that better utilize GPU structure and assets. Finally, we introduce a new family of GPU-dedicated models, called TResNet, which achieve better accuracy and efficiency than previous ConvNets. Using a TResNet model, with similar GPU throughput to ResNet50, we reach 80.8 top-1 accuracy on ImageNet. Our TResNet models also transfer well and achieve state-of-the-art accuracy on competitive single-label classification datasets such as Stanford cars (96.0%), CIFAR-10 (99.0%), CIFAR-100 (91.5%) and Oxford-Flowers (99.1%). They also perform well on multi-label classification and object detection tasks. Implementation is available at: https://github.com/mrT23/TResNet.

1. Introduction

The paper argues that FLOPs are an unreliable proxy for GPU efficiency: newer models can match or exceed ResNet50 accuracy yet deliver worse throughput. TResNet instead targets GPU utilization directly while preserving accuracy.

  • Motivation: FLOPs reductions and architectural tricks often fail to improve GPU throughput, especially during training.The comparison uses Nvidia V100 mixed-precision measurements, with speeds measured at 90% of maximal batch size.
  • Motivation: Modern networks’ extensive use of depthwise and 1x1 convolutions can reduce FLOPs while increasing memory-access costs and fragmenting computation.Multi-path designs also limit the use of inplace operations.
  • Approach: TResNet is designed to improve accuracy while directly optimizing both GPU inference and training speed.TResNet-M is matched to ResNet50 GPU throughput, while larger variants target modern larger architectures.
  • Results: 80.8% ImageNet top-1 accuracy is achieved with GPU throughput similar to ResNet50’s 79.0%.TResNet also reaches 84.3% with better GPU throughput than EfficientNet-B5 at 83.7%.
  • Results: TResNet improves the speed-accuracy trade-off across tested datasets and performs strongly on downstream classification, multi-label classification, and object detection.On three of four downstream single-label datasets, it achieves state-of-the-art accuracy with 8–15× faster GPU inference.

2. TResNet Design

TResNet combines ResNet-derived architectural refinements with GPU-oriented implementation optimizations. Its design targets accuracy, memory efficiency, and throughput through specialized stems, downsampling, blocks, activations, and execution paths.

  • Architecture: TResNet comprises M, L, and XL variants that differ in depth and channel count while retaining a ResNet50-based design.The architecture includes anti-alias downsampling, Inplace-ABN, mixed block types, and optimized SE layers.
  • 2.1. Refinements: The SpaceToDepth stem replaces convolution-based downscaling with a dedicated spatial-to-depth rearrangement that reduces resolution with little information loss.Its purpose is to downscale the input while leaving most processing to residual blocks.
  • 2.1. Refinements: Anti-alias downsampling replaces stride-2 convolutions with stride-1 convolutions followed by a 3x3 blur filter with stride 2.This economical variant is intended to improve the speed-accuracy trade-off.
  • 2.1. Refinements: Inplace-ABN combines BatchNorm and activation in one inplace operation, reducing training memory and enabling larger batch sizes.Leaky-ReLU is used because it improves accuracy over ReLU for TResNet without increasing memory or computational cost.
  • 2.1. Refinements: A mixture of BasicBlock and Bottleneck layers places BasicBlocks in the first two stages and Bottlenecks in the last two.This arrangement is selected for a better speed-accuracy trade-off than uniform block selection.
  • 2.1. Refinements: SE layers are restricted to the first three stages and use stage- and block-specific placements and reduction factors to limit cost.The last stage is excluded because its low-resolution maps receive little accuracy benefit from SE’s global average pooling.
  • 2.2. Code Optimizations: JIT compilation reduces the GPU cost of the AA and SpaceToDepth modules by almost a factor of two.It is applied to parameter-free modules to accelerate execution without imposing limitations.
  • 2.2. Code Optimizations: Inplace operations reduce memory access and unnecessary activation maps, making TResNet-M’s maximal batch size almost twice ResNet50’s, at 512.A dedicated Fast GAP implementation can also be up to five times faster than boilerplate GPU code.

3. ImageNet Results

On ImageNet, TResNet is evaluated against ResNet50 and other architectures using accuracy, GPU throughput, ablations, and higher-resolution fine-tuning. The results show improved speed-accuracy trade-offs, while individual refinements and optimizations expose trade-offs among accuracy, throughput, training speed, and batch size.

  • Basic Training: TResNet models are evaluated on ImageNet at 224-input resolution against other models using top-1 accuracy and GPU throughput.Measurements use Nvidia V100 GPUs with mixed precision; training uses SGD, a 1-cycle policy, and several regularization methods.
  • Basic Training: TResNet-M outperforms the models in the comparison set in both GPU throughput and ImageNet top-1 accuracy.The reported comparison includes newer architectures evaluated for their speed-accuracy trade-off.
  • Basic Training: TResNet models support significantly larger batch sizes than other models, aiding GPU utilization and reducing synchronization between GPUs during distributed learning.The paper also reports that training TResNet-M and ResNet50 takes less than 24 hours on an 8xV100 GPU machine.
  • Network Refinements: SpaceToDepth improves accuracy, inference speed, training speed, and maximal batch size when replacing the conventional convolution-based stem.The authors associate its accuracy improvement with reduced information loss during early downscaling.
  • Network Refinements: Block-type selection using both BasicBlock and Bottleneck blocks improves all measured indices, while fewer third-stage blocks could increase training speed at lower accuracy.Using 11 blocks in the third stage slightly lowers training speed and slightly raises inference speed relative to ResNet50.
  • Network Refinements: Inplace-ABN increases maximal batch size by 200 images, but improves inference speed while somewhat reducing training speed.Optimized SE and anti-aliasing layers improve ImageNet top-1 accuracy at a GPU-throughput cost, which other refinements compensate for.
  • Code Optimizations: Dedicated inplace operations provide the greatest code-optimization boost by improving GPU throughput and maximal batch size.They avoid creating unnecessary activation maps for backward propagation.

4. Transfer Learning Results

TResNet is evaluated through transfer learning on single-label, multi-label, and object-detection tasks. Across these settings, it combines strong accuracy with competitive or improved GPU efficiency.

  • Single-Label Classification: TResNet matches or surpasses state-of-the-art accuracy on 3 of 4 single-label datasets while providing x8-15 faster GPU inference.Results use ImageNet-pretrained checkpoints and single-crop, single-model evaluation.
  • Multi-Label Classification: The multi-label training loss uses separate gamma values for positive and negative samples to address dataset imbalance.This is a variant of focal loss adapted to the multi-label setting.
  • Multi-Label Classification: 86.4 mAP raises the known MS-COCO multi-label classification state of the art from 83.7 mAP.Additional evaluation metrics also improve.
  • Object Detection: TResNet-M increases COCO object-detection mAP from 42.8 to 44.0 compared with ResNet50 at similar GPU throughput.Both models are evaluated within the same FCOS-based detection setup.

5. Conclusion

The paper argues that neural-network design should account for actual GPU utilization, not just FLOPs or inference speed. It proposes GPU-oriented refinements and reports improved throughput-accuracy trade-offs across classification and other vision tasks.

  • Conclusion: GPU training speed and maximal batch size are widely overlooked, although training speed, inference speed, and batch size matter in real-world applications.The paper identifies this measurement gap as a possible blind spot in network design.
  • Conclusion: TResNet combines GPU-oriented refinements, code optimizations, and enhancements into a family of GPU-dedicated models.Listed refinements include SpaceToDepth, economical anti-aliasing downsampling, Inplace-ABN, block-type redesign, and optimized SE layers.
  • Conclusion: Across the ImageNet top-1 accuracy curve, TResNet provides better GPU throughput than existing models.The conclusion also reports new state-of-the-art accuracy on three downstream single-label datasets.
  • Conclusion: TResNet generalizes to multi-label classification and object detection, reaching top scores on those datasets.The conclusion presents these results as evidence beyond single-label ImageNet classification.

A. Code for Different Modules in TResNet

The appendix provides JIT-accelerated implementations for SpaceToDepth and anti-aliasing downsampling, plus a fast global-average-pooling module. These modules reshape, filter, or reduce tensors using GPU-oriented operations.

  • SpaceToDepth: SpaceToDepthJIT reshapes an input tensor from N,C,H,W into N,16C,H/4,W/4 after permuting spatial blocks.The implementation uses a view, permutation, contiguity conversion, and final reshape.
  • Anti-Aliasing Downsampling: AADownsamplingJIT constructs a normalized 3x3 filter, pads inputs reflectively, and applies stride-2 convolution for downsampling.The filter is formed from the one-dimensional coefficients [1, 2, 1].
  • Global Average Pooling: FastGlobalAvgPool2d reduces spatial activations by averaging flattened dimensions, with a specialized implementation for 1x1 outputs.The code supports flattened and explicitly reshaped output forms.
Loading 2003.13630v3…