Source-linked AI summary
HarDNet: A Low Memory Traffic Network
Ping Chao, Chao-Yang Kao, Yu-Shan Ruan, Chien-Hsiang Huang, Youn-Long Lin
TL;DR
CNN model size and MACs do not always predict edge inference latency because intermediate feature-map memory traffic can dominate, particularly in high-resolution applications. The paper proposes CIO, MoC, and HarDNet to reduce this traffic while retaining computational efficiency, reporting lower traffic and inference times than established architectures.
Problem
Edge CNN inference must reduce computation and power use, while model size and MACs may not capture latency dominated by intermediate feature-map DRAM traffic.
Method
The paper introduces CIO as a platform-independent DRAM-traffic approximation and applies soft MoC constraints to design HarDNet with sparse connections and balanced channel widths.
Results
30%∼50% less CIO and 30%∼40% less inference time are achieved at the same accuracy compared with DenseNet and ResNet.
Takeaways & Limitations
Memory traffic for feature maps should be considered alongside model size and MACs when evaluating and designing CNN architectures for edge applications.
Takeaways & Limitations
CIO assumes layer-by-layer CNN execution without fusion and can fail to predict latency across substantially different architectures.
Abstract
from arXiv · showhide
State-of-the-art neural network architectures such as ResNet, MobileNet, and DenseNet have achieved outstanding accuracy over low MACs and small model size counterparts. However, these metrics might not be accurate for predicting the inference time. We suggest that memory traffic for accessing intermediate feature maps can be a factor dominating the inference latency, especially in such tasks as real-time object detection and semantic segmentation of high-resolution video. We propose a Harmonic Densely Connected Network to achieve high efficiency in terms of both low MACs and memory traffic. The new network achieves 35%, 36%, 30%, 32%, and 45% inference time reduction compared with FC-DenseNet-103, DenseNet-264, ResNet-50, ResNet-152, and SSD-VGG, respectively. We use tools including Nvidia profiler and ARM Scale-Sim to measure the memory traffic and verify that the inference latency is indeed proportional to the memory traffic consumption and the proposed network consumes low memory traffic. We conclude that one should take memory traffic into consideration when designing neural network architectures for high-resolution applications at the edge.
1. Introduction
The paper argues that feature-map DRAM traffic can dominate CNN inference latency on edge devices, beyond what model size and MACs indicate. It introduces CIO and MoC to guide architectures that reduce traffic while preserving computational efficiency and accuracy.
- Motivation: DRAM traffic for intermediate feature maps can dominate inference time when memory bandwidth is limited, especially in high-resolution vision tasks.The summed size of intermediate feature maps can be ten to hundred times larger than model size.
- Motivation: Reducing feature-map traffic without accuracy loss requires architectural design rather than lossy compression alone.Quantization and subsampling can reduce feature-map size but usually penalize accuracy.
- Metrics: CIO estimates DRAM traffic by summing the input and output tensor sizes of every convolution layer.For layer l, tensor size is determined by channel count, width, and height.
- Metrics: CIO tracks real DRAM traffic proportionally, but it dominates latency only when a layer’s MACs-over-CIO ratio, or MoC, is sufficiently low.The relevant MoC threshold depends on the hardware platform.
- Architecture: HarDNet applies a soft MoC constraint, sparsifies DenseNet connections to reduce concatenation cost, and balances channel widths across layers.The design avoids layers with very low MoC and accepts a reasonable MAC increase.
- Results: 30%∼50% less CIO and 30%∼40% less inference time are achieved at the same accuracy compared with DenseNet and ResNet.The paper evaluates HarDNet on CamVid, ImageNet, PASCAL VOC, and MS COCO.
2. Related works
Prior work improves CNN efficiency through shortcuts, sparse DenseNet connectivity, and hardware-aware analysis. These approaches motivate connection sparsification but also expose accuracy and traffic trade-offs that HarDNet addresses.
- Shortcut architectures: Residual, Highway, stochastic-depth, and DenseNet architectures use cross-layer shortcuts to support deeper networks and improve information flow.DenseNets concatenate preceding-layer outputs rather than summing them.
- Sparse connectivity: LogDenseNet and SparseNet sparsify DenseNet connections using logarithmic layer links to reduce input-channel growth from O(L^2) to O(L log L).Their block-handling strategies differ for connections crossing resolution blocks.
- Sparse connectivity: Sparse connectivity can reduce accuracy, requiring higher growth rates that may compromise CIO reduction.This trade-off leaves connection sparsification as a promising but incomplete direction.
- Memory-aware efficiency: Prior studies identify limited memory bandwidth as a latency and power constraint across GPU, FPGA, and custom-accelerator systems.The paper presents CIO and MoC as a systematic way to relate DRAM traffic to latency.
3. Proposed Harmonic DenseNet
HarDNet sparsifies DenseNet connections using a power-of-two divisibility rule, then weights key layers and redesigns transitions to reduce CIO while preserving efficiency. Its design includes Harmonic Dense Blocks, optional bottlenecks, and variants using inverted transitions or depthwise-separable convolutions.
- Sparsification and weighting: HarDNet connects layer k to layer k–2^n when 2^n divides k, allowing earlier layers to be flushed after layer 2^n is processed.This power-of-two pattern forms overlapping harmonic waves and reduces concatenation cost relative to LogDenseNet.
- Sparsification and weighting: HarDNet amplifies layers divisible by larger powers of two by increasing their channels, balancing each layer’s input/output channel ratio.The channel multiplier is k × m^n, where n is the maximum power-of-two divisibility exponent and m is a compression factor.
- Implementation illustrations: Figure 2 illustrates DenseNet, LogDenseNet, SparseNet, and HarDNet using 3x3 convolutions, while Figure 3 depicts HarDNet’s inverted transition and depthwise-separable convolution.Table 1 specifies layer, growth-rate, and transition-channel parameters for the implementations.
- Block and bottleneck design: Harmonic Dense Blocks use power-of-two depths, with the final layer having the largest channel count and gradients traversing at most log L layers.A Conv1x1 transition follows each block; bottlenecks may be inserted every four Conv3x3 layers to reduce model size or discarded to improve inference time.
- Transition and bottleneck layers: The inverted transition module adds max pooling alongside average pooling, concatenates both outputs, and then applies Conv1x1.This reduces Conv1x1 CIO by 50% while achieving roughly the same accuracy, at the expense of increased model size.
- Network variants: HarDNet variants include HarDNet-68, which removes global dense connections and uses MaxPool, and HarDNet-39DS, which uses depthwise-separable convolutions.For HarDNet-39DS, point-wise and depth-wise convolution order matters because reversing it increases CIO; CIO alone is not directly comparable when MACs differ greatly.
4. Experiments
Experiments across CamVid, ImageNet, and detection settings show that HarDNet reduces CIO and inference time while maintaining competitive accuracy. Results also support a strong relationship between memory traffic and latency, with platform-dependent differences and specific limitations for small models and simulation.
- CamVid Dataset: 38% less CIO and 24%–36% lower inference time were achieved by FC-HarDNet-ref100 than FC-DenseNet-ref100 on CamVid.The 24% and 36% savings were measured with TensorFlow and PyTorch, respectively, while FC-HarDNet-ref100 used slightly more MACs.
- CamVid Dataset: 41% lower CIO and 35% lower GPU inference time were obtained by FC-HarDNet84 than FC-DenseNet103, while FC-HarDNet68 reduced CIO by 65% and inference time by 52% versus FC-DenseNet56.HarDNet achieved the best accuracy-over-DRAM-traffic among the compared networks, and GPU inference time correlated more with DRAM traffic than MACs.
- CamVid Dataset: 35% lower inference time was maintained by FC-HarDNet-84 than FC-DenseNet-103 in ARM Scale-Sim simulation.The simulation also showed that SegNet’s relative inference time was worse than on GPU, indicating platform-dependent relative DRAM traffic.
- ImageNet Datasets: HarDNet outperformed ResNet and DenseNet on accuracy over CIO, while its accuracy over model size fell between them on ImageNet.For small models, HarDNet did not improve GPU inference time over ResNet, which the authors associated with layer count and concatenation cost.
- ImageNet Datasets: ResNeXt incurred inference-time overhead despite no MAC increase, and the authors attributed that overhead to increased CIO.Table 4 comparisons were sorted by CIO, and CIO remained a rough inference-time predictor across standard-convolution and depth-wise-separable models after weighting.
- Object Detection: HarDNet-68 achieved similar detection accuracy to SSD-ResNet101 while being much faster than VGG-16 and ResNet-101.The model was evaluated as an SSD backbone on PASCAL VOC 2007 and MS COCO, with stride-8 enhancement for small-scale objects.
5. Discussion
The discussion qualifies CIO as a useful but conditional proxy for feature-map traffic and emphasizes balancing memory traffic with computational efficiency. It also argues that traffic matters increasingly as compute capacity outpaces data bandwidth.
- CIO assumptions: CIO assumes layer-by-layer CNN processing without fusion, whereas fused layers can avoid intermediate feature-map traffic.The discussion notes that MobileNetV2’s inverted residual module might be preferable under fused-layer computation.
- CIO limitations: CIO can fail to predict inference time when comparing networks with significantly different architectures.CIO dominates only below a computation-density threshold; otherwise, MACs may remain the key factor.
- CIO limitations: Layer-level inference-latency prediction requires examining each layer’s computation density because CIO and MACs can dominate in different layers.Each network layer has a different MoC, so a whole-network CIO estimate may not capture the latency accurately.
- Hardware implications: As int8 hardware makes computing power grow faster than data bandwidth, data traffic becomes increasingly important for inference efficiency.The discussion contrasts the larger hardware-cost reduction from float32 to int8 multipliers with the smaller reduction in data traffic.
- Design implication: Increasing MoC reasonably is presented as a way to reduce traffic, despite the conventional belief that using more Conv1x1 layers improves efficiency.The paper reports that this approach was helpful in many cases.
6. Conclusion
The paper concludes that feature-map DRAM traffic should complement model size and operation count when evaluating CNN efficiency. HarDNet combines CIO-aware design with higher computational density and achieves substantial traffic and inference-time reductions.
- Conclusion: CIO estimates feature-map DRAM traffic, which can affect system power consumption and, at low computation density, inference time more than model size or operation count.The paper uses CIO to assess accuracy-over-traffic and computational efficiency through MACs over CIO.
- Conclusion: 40% DRAM traffic reduction and 35% GPU inference-time reduction were achieved by FC-HarDNet compared with FC-DenseNet.The reported gains are attributed to the proposed connection pattern and channel balancing.