Source-linked AI summary
Hyperion: An AI-powered HPC cluster for sciences and humanities research that utilizes ML for predicting job turnaround time
Jun Zhou, Nathan Elgar, Tawnee Benedetto, John Richards, Ming Hu, Greg Wilsbacher, Lawrence Miao, Paul Sagona
TL;DR
Researchers need turnaround-time estimates and scalable computation for diverse scientific and humanities workloads. Hyperion addresses both needs through an expandable heterogeneous HPC cluster, Random Forest models for wait and wall time, and Slurm integration, while supporting applications in materials science, neuroimaging, and film research. The models achieved average R-squared values of 0.83 for wall time and 0.74 for wait time.
Problem
Existing turnaround-time estimators are often tailored to other systems and can perform poorly on Hyperion, limiting reliable job planning.
Method
The paper constructs an expandable heterogeneous USC HPC cluster, trains two ML regression models on historical Hyperion jobs, and integrates them into Slurm job submission.
Results
The models achieved average R-squared values of 0.83 for wall time and 0.74 for wait time.
Takeaways & Limitations
Hyperion supports computational research across disciplines while providing users with immediate predicted job turnaround times.
Abstract
from arXiv · showhide
Hyperion is an innovative high-performance computing (HPC) cluster developed for researchers in both science and humanities disciplines at the University of South Carolina (USC). Our approach involved constructing a HPC cluster designed to meet the current research needs while accommodating future expansion. Additionally, we developed and trained two machine learning (ML) models to predict turnaround time, including wait time and wall time, and seamlessly integrated them into the Slurm job submission. Finally, we showcase a variety of sample applications hosted on the Hyperion platform.
1 Introduction
USC consolidated its decentralized research-computing efforts into Hyperion, a heterogeneous HPC resource designed for broad university research needs and future expansion. The paper contributes two turnaround-time prediction models, their Slurm integration, and examples spanning scientific and humanities applications.
- 1 Introduction: USC’s centralized research-computing model reduced redundancy, standardized procedures, improved access, and enabled broader institutional funding.The consolidation under central Information Technology preceded construction of Hyperion as a shared university facility.
- 1 Introduction: Hyperion was built as a heterogeneous, expandable HPC cluster supporting researchers across USC disciplines.Its initial configuration combined CPU, GPU, large-memory nodes, and high-speed interconnects, with later expansions responding to research demand.
- 1 Introduction: Two ML regression models predict wait time and wall time for newly submitted jobs, achieving average R-squared values of 0.74 and 0.83, respectively.The models were integrated into Slurm job-submission scripts so researchers receive turnaround-time predictions.
- 1 Introduction: The paper showcases Hyperion applications from psychology, material sciences, and library sciences.These examples illustrate use of the platform across scientific and humanities-oriented research activities.
2 System Architecture
Hyperion is a centrally operated, heterogeneous Condo-model cluster whose architecture combines CPU, GPU, large-memory, storage, and high-speed networking resources. Its hardware has expanded over time to meet increasing GPU and general compute demand.
- 2 System Architecture: The cluster operates under a Condo model in which hardware purchasers receive subsidized infrastructure while contributing idle cycles to shared queues.USC’s DoIT data center supplies enterprise monitoring, cooling, and backup power for the shared resource.
- 2 System Architecture: Hyperion began with 224 CPU-only nodes, 8 large-memory nodes, and 8 GPU nodes, but GPU demand soon exceeded the initial supply.The original GPU nodes primarily provided a development environment for workloads later scaled on larger national systems.
- 2 System Architecture: Hyperion currently provides 291 CPU nodes, 44 GPU nodes, 8 Big Memory nodes, 16,616 CPU cores, 1.4 PB of GPFS storage, and 100 Gb/s InfiniBand interconnects.Its GPU expansion raised total single-precision performance to 1,232 TFLOPS.
- 2 System Architecture: A May 2023 upgrade added 86 standard CPU nodes and eight 2 TB large-memory nodes while retiring older, less-efficient hardware.The replacement recovered power and network infrastructure capacity for newer compute nodes.
- 2 System Architecture: An NVIDIA DGX A100 added eight 40 GB A100 GPUs and a combined 5 PFLOPS of AI performance to Hyperion.The system was acquired through a partnership between Research Computing and the USC AI Institute.
3 Turnaround Time Prediction
The paper addresses inaccurate, system-specific turnaround-time estimators by training separate wait-time and wall-time models on historical Hyperion jobs. Random Forest models and selected workload features produced the strongest reported predictions and were integrated into Slurm.
- 3 Turnaround Time Prediction: The models used 4.9 million uncleaned jobs from 2018–2023, split into 70% training and 30% testing, with separate regressors for wait and wall time.The records came from 15.3 million Hyperion Slurm jobs and covered 704 users.
- 3 Turnaround Time Prediction: Random Forest performed best among tested regressors, while Submit Time was the most influential feature in both models.Submit Time was normalized to midnight on January 1 of each year before modeling.
- 3 Turnaround Time Prediction: The selected feature set achieved the highest average R-squared for each target: Submit Time, Nodes, Cores, GPUs, Queue ID for wait time, and User ID additionally for wall time.Table 1 reports average outcomes across 2018–2023, with higher R-squared indicating better performance.
- 3 Turnaround Time Prediction: Submit Time ranked above User ID in feature importance, indicating that submission timing strongly influenced turnaround time for Hyperion’s predominantly short jobs.Jobs submitted during off-peak periods or days with fewer paper or grant deadlines tended to execute more promptly.
- 3 Turnaround Time Prediction: The deployed prediction scripts omit User ID despite slightly better wall-time accuracy with it, because USC registers two to three new users weekly.Both models were integrated into Slurm to notify users of predicted turnaround time immediately after submission.
4 Applications
Hyperion supports applications in materials science, neuroimaging, and digital humanities through parallel computation, machine learning, and data-intensive workflows. The examples range from crystal-property prediction and MRI-template construction to film digitization, contextual analysis, and preservation.
- 4.1 Phonon Property Prediction of Crystals: Materials researchers use Hyperion for high-throughput crystal calculations and ML models predicting phonon transport properties.The work combines traditional methods such as random forests with graph neural networks, including Elemental Spatial Density Neural Network Force models.
- 4.2 Neurodevelopmental MRI Database: The Neurodevelopmental MRI Database addresses poor template fit across age groups by adding 3.0T adult scans and refining templates and ancillary materials.The database spans ages from 2 weeks through 89 years, while the new adult sources include nearly 4,600 MRIs from participants aged 20–104.
- 4.2 Neurodevelopmental MRI Database: Hyperion provides parallel computing for refining MRI templates and storing the template database for broader neuroimaging use.The project also targets compatibility with more programs and supports probabilistic projections from scalp locations to stereotaxic cortical areas.
- 4.3 Moving Image Research Collections: MIRC combines film digitization, deep-learning analysis, provenance methods, and a secure collaborative streaming hub for historical-film research.The project had scanned 1,200 films, stored copies in AWS Glacier and a local SAN, and transferred compressed videos to Hyperion for processing.
- 4.3 Moving Image Research Collections: MIRC models detect splices, extra-frame text, and scene text such as ship hull numbers and building nameplates, recording outputs as XML events in Virtual Bench.Two YOLO-based models and an extended textspotter algorithm support the contextual analysis workflow.
5 Conclusion
Hyperion strengthens USC’s research infrastructure through customized HPC capabilities, turnaround-time prediction, and demonstrated use across diverse research domains.
- Hyperion provides computational capabilities across diverse disciplines through customized construction and configuration.
- Two machine learning models predict job turnaround time and help researchers optimize research workflows.
- Three distinct research cases demonstrate Hyperion’s versatility and impact across various domains.