Source-linked AI summary
A Bi-layered Parallel Training Architecture for Large-scale Convolutional Neural Networks
Jianguo Chen, Kenli Li, Kashif Bilal, Xu Zhou, Keqin Li, Philip S. Yu
TL;DR
Large-scale CNN training is time-consuming because it requires many samples and iterative operations. BPT-CNN combines outer-layer distributed parallelism with inner-layer task parallelism, reporting improved training performance while maintaining accuracy.
Problem
Large-scale CNN training requires extensive data and iterative operations, creating a need to accelerate training in distributed computing environments.
Method
BPT-CNN combines outer-layer parallel training across data subsets with inner-layer task parallelism, using IDPA, AGWU, and task decomposition and scheduling.
Results
BPT-CNN effectively improves CNN training performance while addressing data communication, workload balance, and synchronization waiting, without reducing accuracy.
Takeaways & Limitations
BPT-CNN provides a distributed and multi-threaded architecture for accelerating large-scale CNN training while maintaining model accuracy.
Takeaways & Limitations
Future work targets more scalable CNN models, high-performance parallelization, and application-specific deep learning algorithms.
Abstract
from arXiv · showhide
Benefitting from large-scale training datasets and the complex training network, Convolutional Neural Networks (CNNs) are widely applied in various fields with high accuracy. However, the training process of CNNs is very time-consuming, where large amounts of training samples and iterative operations are required to obtain high-quality weight parameters. In this paper, we focus on the time-consuming training process of large-scale CNNs and propose a Bi-layered Parallel Training (BPT-CNN) architecture in distributed computing environments. BPT-CNN consists of two main components: (a) an outer-layer parallel training for multiple CNN subnetworks on separate data subsets, and (b) an inner-layer parallel training for each subnetwork. In the outer-layer parallelism, we address critical issues of distributed and parallel computing, including data communication, synchronization, and workload balance. A heterogeneous-aware Incremental Data Partitioning and Allocation (IDPA) strategy is proposed, where large-scale training datasets are partitioned and allocated to the computing nodes in batches according to their computing power. To minimize the synchronization waiting during the global weight update process, an Asynchronous Global Weight Update (AGWU) strategy is proposed. In the inner-layer parallelism, we further accelerate the training process for each CNN subnetwork on each computer, where computation steps of convolutional layer and the local weight training are parallelized based on task-parallelism. We introduce task decomposition and scheduling strategies with the objectives of thread-level load balancing and minimum waiting time for critical paths. Extensive experimental results indicate that the proposed BPT-CNN effectively improves the training performance of CNNs while maintaining the accuracy.
1 INTRODUCTION
Large-scale CNN training is time-consuming and difficult to distribute because it must coordinate data, synchronization, workload balance, and heterogeneous computing resources. BPT-CNN addresses these challenges with outer- and inner-layer parallelism plus IDPA, AGWU, and task scheduling strategies.
- Motivation: CNN training requires large datasets and many iterative operations, making acceleration critical.The paper frames distributed computing as a way to improve CNN training performance while maintaining accuracy.
- Challenges: Distributed CNN training must coordinate parallel subnetworks while minimizing synchronization waiting and preserving integrated-model accuracy.The challenges include synchronization and integration among subnetworks trained on different machines.
- Challenges: Heterogeneous clusters require dataset partitioning and task allocation that account for differences in node computing speed and capacity.The paper identifies communication overhead, workload balance, and heterogeneous CPU or GPU structures as design constraints.
- BPT-CNN architecture: BPT-CNN combines distributed outer-layer parallelism for data subsets with multi-threaded inner-layer parallelism on each computer.The architecture targets large-scale CNN training by using both cluster-level and computer-level parallel capacity.
- Outer-layer parallelism: IDPA allocates dataset batches according to computing power, while AGWU integrates subnetworks asynchronously to reduce synchronization waiting.These strategies address workload balance, data communication, and global weight updating in the outer layer.
- Inner-layer parallelism: Task decomposition and scheduling parallelize convolutional-layer computation and local weight training to improve thread-level balance and reduce critical-path waiting.The inner-layer strategy applies task-parallelism to two time-consuming CNN training steps.
2 RELATED WORK
Prior CNN acceleration work uses specialized hardware, multi-core platforms, distributed systems, and general-purpose frameworks. BPT-CNN combines distributed-cluster and multi-core CPU parallelism to address communication, synchronization, workload balance, and critical-path waiting.
- Existing approaches: CNN acceleration research has explored FPGAs, many-core architectures, GPUs, distributed systems, and software frameworks.These approaches target higher processing capacity through hardware specialization or distributed computation.
- Distributed systems: DistBelief uses model-level and within-model parallelism with asynchronous SGD for training large neural networks on massive datasets.The paper identifies DistBelief as a distributed architecture relevant to its parallelism design.
- Software frameworks: TensorFlow distributes graph nodes across workers, multi-core CPUs, and GPUs, but the paper notes GPU-memory occupation and customization difficulties.These shortcomings motivate combining ideas from DistBelief and TensorFlow in a new architecture.
- Position of BPT-CNN: BPT-CNN fully uses distributed-cluster and individual-machine multi-core parallel capacity.Its optimization targets include data communication, workload balancing, and critical-path waiting.
3 BPT-CNN ARCHITECTURE FOR CNNS
BPT-CNN accelerates large-scale CNN training through outer-layer data parallelism across distributed computers and inner-layer task parallelism within each computer. Its design addresses workload balance, communication overhead, and synchronization waiting in heterogeneous clusters.
- Motivation: CNNs achieve high accuracy but require substantial computation because complex networks use massive weight parameters, large training datasets, and iterative weight updates.These requirements can bottleneck training when available computing power is insufficient.
- Architecture: BPT-CNN uses outer-layer data parallelism to train CNN subnetworks on separate data subsets while a parameter server updates and shares global weights.The dataset is split across computing nodes, and local weights contribute to successive global updates.
- Architecture: Inner-layer task parallelism decomposes convolutional computation and local weight training for multithreaded execution within each computing node.Tasks are decomposed according to logical and data dependencies.
- Distributed environment: The distributed cluster combines a main server, heterogeneous mult-core computing nodes, and a parameter server for task management, data allocation, local training, and global updates.The main server monitors execution times and can migrate datasets to reduce synchronization delay.
- Outer-layer optimization: IDPA allocates large-scale training data incrementally according to node capability, enabling similar iteration durations, workload balance, and no data migration during training.The stated result is reduced synchronization delay and avoided unnecessary communication overhead.
- Outer-layer optimization: AGWU updates the global weights whenever a node finishes local training, eliminating synchronization waiting without increasing communication overhead.Each node independently submits local weights to the parameter server, which immediately produces a new global version.
4 INNER-LAYER PARALLEL TRAINING OF BPT-CNN
BPT-CNN parallelizes each CNN subnetwork within a computing node by decomposing convolution and local weight-training computations into concurrently scheduled tasks. The design targets thread-level load balancing and reduced critical-path waiting.
- Overview: Inner-layer parallelism accelerates each CNN subnetwork by parallelizing convolutional-layer computation and local weight training on each computer.Tasks are decomposed according to logical and data dependence and executed with multi-threaded parallelism.
- Parallelization of Convolutional Layer: Convolutional operations partition the input matrix into convolution areas and process them concurrently with a shared filter matrix.Each task multiplies one extracted input area element-by-element with the filter to produce a feature-map element.
- Parallelization of Convolutional Layer: The maximum convolution parallelism degree equals the number of output feature-map elements.The convolution-layer duration is calculated from the execution of these parallel operations.
- Parallelization of Local Weight Training Process: Local weight training parallelizes loss-error computation across neurons and computes weight gradients and updates concurrently.Errors are propagated from the output layer toward earlier layers before gradients and weights are processed.
- Task Decomposition and Scheduling: Task decomposition creates subtasks for the CNN subnetwork, while DAG-based priority marking supports thread-level load balancing and completion-time minimization.The workflow is illustrated as a task DAG whose priorities reflect logical and data dependence.
5 EXPERIMENTS
Experiments on a 30-node distributed cluster compare BPT-CNN with TensorFlow, DisBelief, and DC-CNN across accuracy, execution time, strategy choices, communication, and workload balance. BPT-CNN maintains comparable accuracy while generally improving performance, especially at larger resource scales.
- Accuracy Evaluation: BPT-CNN achieves average accuracy of 0.744, versus 0.721 for TensorFlow, 0.722 for DisBelief, and 0.639 for DC-CNN.Its AUC is also higher on average than TensorFlow, DisBelief, and DC-CNN by 5.91%, 9.56%, and 10.09%, respectively.
- Execution Time Comparison: BPT-CNN achieves higher performance than comparison algorithms in most tested data-size and cluster-scale configurations.The experiments use 100 training iterations and report high performance and scalability in distributed computing clusters.
- Execution Time Comparison for Fixed Accuracy: BPT-CNN requires 42 iterations to reach accuracy 0.750, compared with 64 for TensorFlow, 85 for DisBelief, and 147 for DC-CNN.The comparison evaluates the iterations and execution times needed to reach fixed accuracy under varying computing resources.
- Strategy Comparison: AGWU usually reduces execution time relative to SGWU, while IDPA keeps computing-node workloads better balanced than UDPA.Asynchronous global updates reduce waiting, and incremental allocation accounts for heterogeneous node computing power.
- Communication and Workload Balancing: BPT-CNN combines lower data-communication costs with stable workload balance between 0.89 and 0.80 as cluster size increases from 5 to 35 nodes.IDPA avoids training-sample migration and allocates work according to each node’s actual computing power.
6 CONCLUSIONS
The paper presents BPT-CNN as a bi-layered architecture for accelerating large-scale CNN training. Experiments indicate improved training performance with minimal data communication and synchronization waiting, while future work targets scalable CNN models and high-performance computing.
- BPT-CNN accelerates large-scale CNN training through bi-layered parallelism.The architecture combines outer-layer and inner-layer parallel training.
- Outer-layer data parallelism addresses data communication, workload balance, and synchronization across distributed computers.
- Inner-layer task parallelism further accelerates training for each CNN subnetwork.
- Extensive experiments indicate improved CNN training performance with minimum data communication and synchronization waiting.
- Future work focuses on scalable CNN models and parallelizing deep learning algorithms on high-performance computers.