Source-linked AI summary
DPP-Net: Device-aware Progressive Search for Pareto-optimal Neural Architectures
Jin-Dong Dong, An-Chieh Cheng, Da-Cheng Juan, Wei Wei, Min Sun
TL;DR
NAS commonly neglects device-related objectives such as latency, despite their importance for practical deployment. DPP-Net performs progressive, device-aware multi-objective search over a compact mobile-CNN-inspired space, finding Pareto-optimal architectures with higher accuracy and shorter inference time across devices and strong ImageNet performance.
Problem
Most NAS methods optimize a single objective such as accuracy while largely ignoring device-related objectives, leaving joint optimization of accuracy and device constraints underaddressed.
Method
DPP-Net efficiently searches Pareto-optimal architectures across device-related and device-agnostic objectives using a compact mobile-CNN-inspired search space and progressive search.
Results
DPP-Net finds Pareto-optimal networks across three devices, achieves higher accuracy and shorter inference time than CondenseNet, and outperforms NASNet (Mobile) across ImageNet metrics.
Takeaways & Limitations
The results support device-aware Pareto search as a way to expose architecture trade-offs for selecting models under specific deployment conditions.
Abstract
from arXiv · showhide
Recent breakthroughs in Neural Architectural Search (NAS) have achieved state-of-the-art performances in applications such as image classification and language modeling. However, these techniques typically ignore device-related objectives such as inference time, memory usage, and power consumption. Optimizing neural architecture for device-related objectives is immensely crucial for deploying deep networks on portable devices with limited computing resources. We propose DPP-Net: Device-aware Progressive Search for Pareto-optimal Neural Architectures, optimizing for both device-related (e.g., inference time and memory usage) and device-agnostic (e.g., accuracy and model size) objectives. DPP-Net employs a compact search space inspired by current state-of-the-art mobile CNNs, and further improves search efficiency by adopting progressive search (Liu et al. 2017). Experimental results on CIFAR-10 are poised to demonstrate the effectiveness of Pareto-optimal networks found by DPP-Net, for three different devices: (1) a workstation with Titan X GPU, (2) NVIDIA Jetson TX1 embedded system, and (3) mobile phone with ARM Cortex-A53. Compared to CondenseNet and NASNet (Mobile), DPP-Net achieves better performances: higher accuracy and shorter inference time on various devices. Additional experimental results show that models found by DPP-Net also achieve considerably-good performance on ImageNet as well.
1 Introduction
NAS methods often optimize accuracy while neglecting device-related objectives, making multi-objective, device-constrained architecture search an important open problem. DPP-Net addresses this by efficiently searching Pareto-optimal architectures and reports strong results across devices and datasets.
- Most NAS approaches optimize a single objective, such as accuracy, while largely ignoring device-related objectives including latency.
- Jointly searching for high accuracy, inference time, model size, and other device constraints remains a critical but underaddressed question.
- DPP-Net searches Pareto-front architectures across device-related and device-agnostic objectives, enabling practitioners to select models for specific use cases.
- DPP-Net finds Pareto-optimal networks on a Titan X workstation, Jetson TX1, and ARM Cortex-A53 mobile phone.
- DPP-Net achieves higher accuracy and shorter inference time than CondenseNet across the three evaluated devices, with additional strong ImageNet performance.
2 Related Work
Prior NAS research is organized around reinforcement learning, genetic algorithms, and other search strategies, while multi-objective and handcrafted efficient CNNs address efficiency through different approaches. The related work highlights a gap between automated search and practical device-aware optimization.
- NAS methods are commonly categorized as reinforcement-learning-based, genetic-algorithm-based, or based on other optimization techniques.
- RL-based approach: RL-based NAS uses a controller to generate architectures and evaluates validation accuracy as the controller’s reward; NASNet improves this with PPO and repeated block search.
- GA-based approach: Genetic-algorithm methods are another major NAS family, with recent work achieving strong CIFAR-10 performance compared with RL-based methods.
- Other approaches: Alternative strategies such as MCTS and SMBO aim to reduce the computational burden of architecture search.
- Architecture search with multiple objectives: Multi-objective NAS has considered runtime speed and classification accuracy, but reported performance was not comparable to handcrafted models.
- Handcrafted models with multiple objectives: Efficient handcrafted CNNs use operations such as depth-wise convolution and ShuffleNet-style designs to reduce computation while retaining comparable accuracy.
3 Search Architecture
DPP-Net fixes the overall network scaffold before search and optimizes only the repeated Dense Cell structure. The architecture differs between CIFAR-10 and ImageNet through cell repetitions, growth rate, and input-scale-specific settings.
- DPP-Net repeats an identical Dense Cell following CondenseNet’s connecting rules, with dense feature-map connections and a growth rate that doubles when resolution decreases.
- The overall architecture is fixed before search, leaving the cell structure as the only component being optimized.
- CIFAR-10 and ImageNet architectures use different Dense Cell repetition counts and growth rates, while ImageNet uses initial-convolution stride 2 and global-pooling size 7.
- Each searched cell alternates normalization and convolution layers in a progressively expanded Norm-Conv-Norm-Conv sequence.
- Normalization choices include Batch Normalization with ReLU, Batch Normalization, and identity operations.
4 Search Space
The search space combines efficient convolutional operations drawn from mobile CNN design with a compact alternating cell structure. Its combinatorial size grows by multiplying the available normalization and convolution choices at each layer.
- Convolution choices include standard, group, learned-group, and depth-wise convolutions with 1x1 or 3x3 kernels.
- The search space incorporates efficient operations such as depth-wise and learned-group convolution to use empirical mobile-CNN knowledge and reduce search time.
- DPP-Net’s cell design is presented alongside cells from other efficient CNNs, with group convolutions followed implicitly by channel shuffle.
- For an ℓ-layer cell, the number of possible structures is the product of available operations at each layer, alternating normalization and convolution choices.
- The stated operation counts yield 54 possible three-layer cells and 324 possible four-layer cells.
5 Search Algorithm
DPP-Net uses progressive Sequential Model-Based Optimization to search a growing architecture space efficiently, then applies constraints and Pareto optimality to retain trade-off solutions.
- Progressive Search: The search progressively expands architectures from smaller to larger spaces instead of evaluating the complete space directly.This reduces the burden of training every possible architecture as the number of layers increases.
- Search Procedure: The algorithm has three stages: Train and Mutation, Update and Inference, and Model Selection.During mutation, each ℓ-layer model generates ℓ+1-layer candidates by exploring possible operation combinations.
- Surrogate Modeling: A surrogate function predicts the accuracies of mutated candidates using architectures and measured accuracies from previously trained models.This avoids time-consuming training of every candidate network.
- Model Selection: Hard constraints remove models that fail minimum requirements, while soft constraints become objectives for Pareto-optimal selection.Pareto optimality retains solutions where improving one objective would worsen another.
- Surrogate Modeling: The surrogate is implemented as an RNN that handles variable-length architecture inputs and regresses accuracy through a sigmoid-activated fully connected layer.Cell structures are represented using one-hot encodings and learned embeddings.
6 Experiments and Results
DPP-Net searches device-aware Pareto-optimal architectures across multiple objectives and devices, then evaluates selected models on CIFAR-10 and ImageNet. Results show trade-offs among accuracy, complexity, and device-specific inference time, with DPP-Net outperforming cited baselines in key comparisons.
- Experimental Details: DPP-Net searches Pareto-optimal architectures using error rate, parameters, FLOPs, and device-specific inference time, adding memory usage for mobile-phone searches.The workstation and embedded-system searches use four objectives, while the mobile-phone search uses five.
- Results on CIFAR-10: Fig. 7 presents Pareto-front candidates trained for 10 epochs on CIFAR-10, comparing error rate with parameters, FLOPs, and actual inference time against CondenseNet.Candidates are color-coded by DPP-Net search setting, and each represents a Pareto-optimal architecture.
- Results on CIFAR-10: Inference time cannot be simply inferred from parameters or FLOPs because hardware implementation and device characteristics affect runtime.The paper specifically notes that depth-wise and group convolutions may fail to achieve their theoretical speedups in practice.
- Results on CIFAR-10: The surrogate function has a best validation regression error of around 12% and was used to identify relatively promising architectures rather than predict exact accuracy.The search used four GTX 1080 GPUs for around 48 hours.
- Results on ImageNet: DPP-Net-Panacea outperforms CondenseNet-74 and NASNet (Mobile) on nearly every reported ImageNet metric.Mobile-device inference time and memory usage could not be measured for NASNet (Mobile) because required operations were unsupported in the deployment framework.
7 Conclusions
DPP-Net is presented as a device-aware NAS approach that outperforms handcrafted mobile CNNs, with Pareto-optimal networks effective across three devices and strong ImageNet performance.
- DPP-Net is described as the first device-aware NAS approach to outperform state-of-the-art handcrafted mobile CNNs.
- On CIFAR-10, DPP-Net finds effective Pareto-optimal networks across a workstation GPU, an embedded Jetson TX1, and an ARM Cortex-A53 phone.
- Compared with CondenseNet and NASNet (Mobile), DPP-Net achieves higher accuracy and shorter inference time across these devices.
- Models found by DPP-Net also achieve state-of-the-art performance on ImageNet.