Source-linked AI summary
Deploying a Top-100 Supercomputer for Large Parallel Workloads: the Niagara Supercomputer
Marcelo Ponce, Ramses van Zon, Scott Northrup, Daniel Gruner, Joseph Chen, Fatih Ertinaz, Alexey Fedoseev, Leslie Groer, Fei Mao, Bruno C. Mundim, Mike Nolta, Jaime Pinto, Marco Saldarriaga, Vladimir Slavnic, Erik Spence, Ching-Hsing Yu, W. Richard Peltier
TL;DR
The paper addresses how to design and deploy a large parallel supercomputer while transitioning users from previous systems. It describes Niagara’s procurement, deployment, configuration, tools, and validation process. The resulting 60,000-core cluster achieved 3.02 petaflops measured HPL performance and debuted at #53 in the June 2018 TOP500 list.
Problem
Niagara had to support large parallel jobs while satisfying competing requirements for performance, transition continuity, and system operation.
Method
The paper describes Niagara’s design, procurement, deployment, installation, testing, configuration, user tools, scheduling, and early science validation.
Results
3.02 petaflops of measured HPL performance placed Niagara at #53 in the June 2018 TOP500 list.
Takeaways & Limitations
Niagara combines homogeneous hardware, a DragonFly+ interconnect, high-speed burst-buffer storage, Cooperage, and user-facing monitoring tools.
Takeaways & Limitations
Burst-buffer file persistence is very limited, so users should stage final files to /scratch and remove temporary files after each job.
Abstract
from arXiv · showhide
Niagara is currently the fastest supercomputer accessible to academics in Canada. It was deployed at the beginning of 2018 and has been serving the research community ever since. This homogeneous 60,000-core cluster, owned by the University of Toronto and operated by SciNet, was intended to enable large parallel jobs and has a measured performance of 3.02 petaflops, debuting at #53 in the June 2018 TOP500 list. It was designed to optimize throughput of a range of scientific codes running at scale, energy efficiency, and network and storage performance and capacity. It replaced two systems that SciNet operated for over 8 years, the Tightly Coupled System (TCS) and the General Purpose Cluster (GPC). In this paper we describe the transition process from these two systems, the procurement and deployment processes, as well as the unique features that make Niagara a one-of-a-kind machine in Canada.
1 INTRODUCTION
The paper presents Niagara’s design, procurement, deployment, testing, and setup, emphasizing practices for a state-of-the-art homogeneous cluster and early full-system science runs.
- The paper describes Niagara’s design, procurement, deployment, installation, testing, and setup.
- It focuses on best practices developed for a homogeneous cluster that could benefit others undertaking similar projects.
- Early science cases used heroic full-system runs to test, validate, and fine-tune Niagara.
- These runs had already produced scientific results, including one receiving the 2018 HPCwire Editor’s Award for Best Use of HPC in Physical Sciences.
2 REQUIREMENTS AND DESIGN
Niagara’s design integrated competing requirements for large parallel workloads, continuity from previous systems, national compatibility, and practical operational constraints.
- Requirements and design: Niagara was designed as a machine dedicated to large parallel jobs with intense computational and communication requirements.
- Requirements and design: Consultations found that major users preferred a homogeneous CPU cluster with a fast interconnect over hybrid or pure-GPU systems.
- Smooth Transition from Previous Systems: The transition was planned to minimize disruption and interruption while replacing the heavily used GPC and TCS systems.
- Requirements and design: Design choices were informed by more than a decade of cluster experience, including hardware, memory, interconnect, storage, directory, and scheduling considerations.
- Compatibility and Coordination with Other Canadian National Systems: Niagara was required to coordinate with Canada’s national computational platform while supporting national authentication and local users.
- Requirements and design: The design also incorporated a budgetary constraint of 20 million Canadian dollars, approximately 15 million USD.
3 HARDWARE SELECTION
Hardware selection combined representative large-parallel benchmarks with technical and operational evaluation, producing a 60,000-core system designed for scalable computation, communication, and storage.
- Hardware selection: The Large Parallel Benchmark comprised HPCG, Nek5000, WRF, NAMD, miniDFT, SPEC MPI 2007, and IOR.
- Hardware selection: The LPBM normalized benchmark results against the GPC and produced a single throughput and performance-improvement score.
- Proposals Evaluation: 11 proposals from 6 vendors were evaluated; Lenovo was selected, with contract signing in October 2017 and full system acceptance in March 2018.
- Resulting Hardware Specifications: 3.02 petaflops of measured HPL performance placed Niagara at #53 in the June 2018 TOP500 list, versus 4.6 petaflops theoretical Rmax.
- Resulting Hardware Specifications: The cluster contains 1500 Lenovo SD530 servers, each with 40 Intel Xeon Gold 6148 Skylake cores at 2.4 GHz.
- Resulting Hardware Specifications: Niagara provides approximately 9PB of usable shared parallel storage from 12PB raw capacity and a 256TB burst buffer supporting up to 160 GB/s.
- Deployment: A four-node Test and Development System using the proposed compute-node hardware supported configuration and software-stack testing before and during production.
4 SOFTWARE STACK
Niagara uses a modular software environment supported by Cooperage, a cluster-specific build system that enables systematic maintenance and multiple application versions without conflicts.
- Software stack: The module-based software stack allows multiple versions of the same application to be installed without conflicts.
- Software stack: Cooperage systematically builds, maintains, documents, rebuilds, reinstalls, and upgrades software modules.
- Software stack: Cooperage was developed after the previous unsystematic installation process caused maintenance headaches.
- Software stack: The tool is simpler and less generic than alternatives such as EasyBuild and Spack, with an approach tailored to Niagara.
- Software stack: Niagara offers native and general-purpose software stacks, switchable through the same Lmod module system.
- Software stack: The general-purpose stack runs without node-local disks by using CVMFS alien cache technology and GPFS performance.
5 JOB SCHEDULER
Niagara uses SLURM to allocate jobs through a deliberately simple, fair-share-oriented scheduling configuration. Its scheduler supports varied partitions and job policies while accounting for hyperthreading and cluster-specific constraints.
- SLURM tracks current and future resources and allocates them to queued jobs according to priority.
- The scheduler began with minimal constraints, adding complexity only when usage data showed significant deviations from fair usage.
- Jobs default to the compute partition, request 1–1000 nodes, and run for 15 minutes to 24 hours.
- Archive partitions provide differentiated limits for tape transfers, recalls, and interactive HPSS usage.
- The multifactor priority plugin emphasizes fair-share while also considering job age, size, partition, and quality of service.
- Hyperthreading exposes twice as many logical cores, and CPU-bound applications typically gain 5–10% speedup.
6 FILE SYSTEMS
Niagara combines GPFS-based shared storage with scratch, project, home, archive, and burst-buffer spaces tailored to different data lifecycles and I/O patterns. Its hierarchical directory layout improves organization, permissions, and parallel file-system responsiveness.
- GPFS provides rapid parallel reads and writes, while Niagara offers several distinct file systems for different storage needs.
- /home stores individual files, shared software, and small group datasets, whereas /scratch stores temporary or reproducible computation outputs.
- /project stores shared software and costly-to-recreate datasets with backup, while /bb provides a faster SSD alternative to scratch.
- The burst buffer is optimized for high I/O operations per second, including bioinformatics, quantum chemistry, and large restart checkpoints.
- Persistent burst-buffer files survive between jobs, but persistence is limited and users should stage final files to /scratch and remove temporary data.
- Hierarchical paths organize users by group and first letter, improving group permissions and parallel file-system responsiveness over flat directories.
7 MONITORING TOOLS
Niagara provides web and command-line tools for monitoring system, job, resource, and allocation utilization. These tools also support analysts in identifying inefficient usage and evaluating how fairly resources are allocated.
- 7 MONITORING TOOLS: System-specific tools let users track Niagara’s system and resource utilization through a web interface and command-line tools.The tools were developed specifically for Niagara and are offered through my.scinet.utoronto.ca and command-line utilities.
- 7.1 my.scinet Portal: The my.scinet portal provides real-time views of system status, storage, job states, turnover times, resource usage, and individual job performance.Authenticated users can also inspect allocations, storage utilization, past jobs, and metrics including core utilization, communication, IOPs, and FLOPS.
- 7.1 my.scinet Portal: The portal’s job-performance data helps analysts identify less than optimal system usage.The same information is available to users and supports operational analysis of system utilization.
- 7.2 Command-line Tools: qsum summarizes queued and running jobs by user, including cores, job counts, and estimated durations.It provides a user-centered alternative to inspecting jobs individually with SLURM’s squeue command.
- 7.2 Command-line Tools: The command-line tools help evaluate what fraction of the system is being used by default users and how fairly resources are allocated.Additional utilities query job performance, node performance, filesystem utilization, allocation, and scheduling priority.
8 EARLY SCIENCE CASES
After deployment, selected scientists ran near-full-system “heroic” calculations to test, tune, and prepare Niagara for national academic production. Four early science cases covered ocean circulation, stellar hydrodynamics, and numerical relativity among other large-scale simulations.
- 8 EARLY SCIENCE CASES: Selected scientists used nearly the entire system for “heroic” calculations immediately after deployment.These runs were intended to test, tune, and prepare Niagara for use as Canada’s fastest national academic supercomputer.
- 8 EARLY SCIENCE CASES: Four early science cases addressed global ocean circulation, stellar hydrodynamics, numerical relativity, and another large-scale scientific problem.The stellar case simulated core convection in a 25-solar-mass star on a 1536-cubed grid over 57.7 days of star time.
9 TRANSITION AND MIGRATION
Niagara’s transition was planned to minimize disruption from two highly utilized predecessor systems. File migration and account transfer moved substantial existing activity while adding new users.
- 9 TRANSITION AND MIGRATION: Because the predecessor systems were highly utilized, SciNet prioritized a smooth transition to Niagara.The deployment was designed to minimize disruption and interruption of users’ work.
- 9 TRANSITION AND MIGRATION: SciNet migrated users’ files and accounts through repeated rsync transfers, a login freeze, and a final integrity check.The process began several weeks before GPC shutdown and included migration from both GPC and TCS.
- 9 TRANSITION AND MIGRATION: 2216 of 2627 GPC accounts were migrated to Niagara.This preserves the reported account counts from the decommissioning process.
- 9 TRANSITION AND MIGRATION: 2537 of 3104 Niagara accounts were enabled, and approximately 35% of current accounts were new relative to GPC.Among the active accounts, 1690 had previously been active on GPC.
10 CONCLUSION
The paper presents Niagara’s procurement, deployment, architecture, storage, software, and user-support tools as an integrated platform for large parallel workloads. Near-maximum capacity and more than 1.2 million jobs indicate strong demand among Canadian researchers.
- 10 CONCLUSION: Niagara combines homogeneous cluster design, a DragonFly+ high-speed interconnect, a high-IOPS burst buffer, the Cooperage build system, and customized user tools.The user tools are made available to the community through a public repository.
- 10 CONCLUSION: Niagara operated near maximum capacity from general production and surpassed 1.2 million submitted jobs in slightly over a year.The paper presents these observations as an indication of Canadian researchers’ need for large-parallel-job capability.
A SCINET/NIAGARA ONLINE RESOURCES
SciNet provides separate online resources for its main website, education and training, system status, technical documentation, system statistics, and Niagara’s deployment timeline.
- SciNet’s online resources include its main website and education and training materials.
- Technical resources cover system status and technical documentation.
- Additional resources provide system statistics and Niagara’s deployment timeline.