Source-linked AI summary
'Zhores' -- Petaflops supercomputer for data-driven modeling, machine learning and artificial intelligence installed in Skolkovo Institute of Science and Technology
Igor Zacharov, Rinat Arslanov, Maxim Gunin, Daniil Stefonishin, Sergey Pavlov, Oleg Panarin, Anton Maliutin, Sergey Rykovanov, Maxim Fedorov
TL;DR
Scientific users need computing resources that accommodate multidisciplinary workloads across data-driven modeling, machine learning, Big Data, and AI, with demands ranging from throughput to capability computing. The paper presents Zhores’ implementation and evaluates its processors, GPUs, network, applications, and usage modes through benchmarks and practical deployments. Initial tests report scalable modeling applications, a 2.5× GPU-over-CPU performance advantage for Gromacs, and resources sufficient to start a rapidly expanded hackathon on time.
Problem
Skoltech requires computing resources for multidisciplinary data-driven modeling, machine learning, Big Data, and AI workloads with heterogeneous throughput, capability, and data-analysis demands.
Method
The paper presents Zhores’ cluster implementation and evaluates its CPUs, GPUs, network, applications, scaling, and usage modes using benchmarks and practical deployments.
Results
Initial tests show good modeling-application scalability; Gromacs with four GPUs per node achieves about 2.5 times CPU-only performance, while virtualization supports all hackathon teams.
Takeaways & Limitations
Zhores can support advanced multidisciplinary research at Skoltech and for its research and industrial partners.
Abstract
from arXiv · showhide
The Petaflops supercomputer "Zhores" recently launched in the "Center for Computational and Data-Intensive Science and Engineering" (CDISE) of Skolkovo Institute of Science and Technology (Skoltech) opens up new exciting opportunities for scientific discoveries in the institute especially in the areas of data-driven modeling, machine learning and artificial intelligence. This supercomputer utilizes the latest generation of Intel and NVidia processors to provide resources for the most compute intensive tasks of the Skoltech scientists working in digital pharma, predictive analytics, photonics, material science, image processing, plasma physics and many more. Currently it places 6th in the Russian and CIS TOP-50 (2018) supercomputer list. In this article we summarize the cluster properties and discuss the measured performance and usage modes of this scientific instrument in Skoltech.
1. Introduction
“Zhores” is a petaflops cluster designed for multidisciplinary research spanning HPC, Big Data, and AI. Its architecture addresses heterogeneous computing demands, and the paper documents its implementation, benchmarks, applications, and usage.
- “Zhores” supports data-driven simulations and modeling, machine learning, Big Data, and artificial intelligence research at Skoltech.
- Its architecture converges traditional HPC, Big Data, and AI to serve workloads ranging from throughput computing to capability computing.
- Heterogeneous project requirements also motivate workflow acceleration and in-situ data analysis capabilities.
- The paper describes the cluster’s implementation and initial benchmarks, then examines applications, scaling, and a Neurohackathon deployment.
2. Installation
“Zhores” combines CPU and GPU compute nodes, high-speed storage and Mellanox EDR InfiniBand into a managed heterogeneous cluster. Measurements characterize processor, memory, GPU, interconnect and startup performance, including power-efficiency and network-contention trade-offs.
- Architecture: “Zhores” uses DELL PowerEdge C6400 and C4140 servers with Intel Xeon CPUs, Nvidia Volta GPUs, Mellanox EDR InfiniBand, 20 TB NVMe-over-IB storage and 0.6 PB GPFS storage.
- Architecture: CPU, GPU, Hadoop, access, experiment, visualization and management nodes divide the cluster’s computing, storage, user-access and administrative functions.
- CPU and memory: 150 TFlop/s is the measured maximum CPU performance across all “cn” and “gn” nodes, with 2.0 TFlop/s on C6140 nodes and 2.4 TFlop/s on C4140 nodes.
- CPU and memory: 178.6 GB/s is the measured total STREAM Triad memory bandwidth using all six channels of 2666 MHz DIMMs.
- CPU and memory: FMA performance scales strongly with processor frequency, whereas memory bandwidth depends weakly on frequency, supporting power-usage optimization for mixed instruction profiles.
- GPU subsystem: GPU nodes contain four pairwise-NVLinked Nvidia V100 GPUs, while cluster-network traffic passes through a single PCIe link; V100 efficiency peaks at 67.4 GFlop/s/W at 1 GHz.
- Network and management: Serial interconnect communication centers on 10.2 ± 0.5 GB/s, while parallel traffic exposes Fat Tree oversubscription caused by additional file-service traffic.
- Network and management: A single node boots within 2.5 minutes and the complete cluster reaches operational state after a 5-minute cold start; “Zhores” ranks sixth in the Russian and CIS TOP-50.
3. Applications
Applications on “Zhores” benchmarked aggregation, coagulation, and molecular-dynamics solvers across CPU and GPU resources. Results show strong CPU scalability for advection-driven coagulation and substantial GPU gains for Gromacs, while aggregation–fragmentation scaling is limited by ClusterFFT.
- Aggregation and fragmentation: More than 85× speedup was obtained on 128 cores for the aggregation–fragmentation solver, despite limited scalability from the dominant ClusterFFT operation.The benchmark used parallel numerical methods for aggregation and fragmentation equations.
- Advection-driven coagulation: Advection-driven coagulation achieved almost ideal acceleration across nearly the full CPU-based segment through one-dimensional spatial domain decomposition.The experiments used Intel compilers and the Intel MKL library.
- Ternary aggregation: Ternary aggregation was benchmarked alongside binary aggregation, with higher floating-point and memory demands than the binary case.The ternary benchmark used 10 time-integration steps for N = 219 nonlinear ODEs.
- Molecular dynamics: 125 million Lennard-Jones spheres were simulated with Gromacs 2018.3 using a 1.2 nm cutoff radius, a Berendsen thermostat, and single precision.Performance was measured over up to 40 CPU nodes and 24 GPU nodes, averaging five runs per task.
- Molecular dynamics: The CPU runs showed almost ideal speedup before deviating from linear growth at larger node counts.GPU tests compared configurations using 1, 2, and 4 graphics cards per node.
- Molecular dynamics: 4 GPUs per node delivered about 2.5 times the performance of a CPU-only run on 24 GPU nodes, while 1 GPU per node still improved performance significantly.Using all four GPUs per node showed good scalability; two GPUs per node showed slightly lower speedup, and one GPU per node performed worst at high node counts.
4. Neurohackathon at Skoltech
“Zhores” provided the main computing infrastructure for a neuro-medicine hackathon analyzing private MRI data related to multiple sclerosis. Virtualization, network isolation, and authenticated access enabled simultaneous participation while protecting the computing environment.
- Hackathon scope: The November 2018 Neurohackathon included scientific and open tracks, with the scientific track analyzing private MRI images showing changes characteristic of multiple sclerosis.The event was held at Skoltech from 16 to 18 November 2018 under the National Technology Initiative.
- Access and security: Participants accessed the cluster through a two-stage system using VPN authentication followed by SSH access to operating-system consoles.Cisco ASA and Cisco AnyConnect provided the VPN connection.
- Access and security: VLAN-based network virtualization and Docker containers isolated participant environments, with namespaces and remapped user and group IDs further separating users from the host.Containers could connect to GPU accelerators while operating in separate address spaces and VLANs.
- Computing environment: Each participant received a Docker container with SSH console access and HTTPS access to the Jupyter application on a virtual machine.Four Nvidia Tesla V100 accelerators on GPU nodes provided computing resources.
- Operational outcome: The number of participating teams increased from 6 to 11 one hour before the event, yet virtualization and the cluster’s flexible architecture supplied resources and allowed the hackathon to start on time.The infrastructure supported all teams despite the late increase in participation.
5. Conclusions
The article presents “Zhores” as a Petaflops Skoltech supercomputer for multidisciplinary research in data-driven simulations, machine learning, Big Data, and artificial intelligence. Its Linpack ranking and initial application tests indicate broad intended use and good scalability.
- Conclusions: “Zhores” is intended for multidisciplinary research in data-driven simulations, machine learning, Big Data, and artificial intelligence.The system is installed in Skoltech CDISE.
- Conclusions: Linpack placed “Zhores” at position 6 in the Russian and CIS TOP-50 Supercomputer list.The ranking is reported as part of the concluding assessment of the cluster.
- Conclusions: Initial tests showed good scalability of the modeling applications and support the cluster’s use for advanced research at Skoltech and by its research and industrial partners.The conclusion frames the cluster as a new computing instrument for these users.