Source-linked AI summary
Scientific Machine Learning Benchmarks
Jeyan Thiyagalingam, Mallikarjun Shankar, Geoffrey Fox, Tony Hey
TL;DR
Scientific ML must help analyze rapidly growing experimental datasets, yet choosing suitable algorithms, architectures, and frameworks remains challenging. The paper develops a benchmarking approach and reviews initiatives, arguing for benchmark suites that support users and new benchmark development while noting unresolved dataset-distribution concerns.
Problem
Scientists face a difficult choice among ML algorithms, computer architectures, and frameworks for analyzing large scientific datasets.
Method
The paper defines scientific ML benchmarks, develops guidelines and best practices, and reviews benchmarking initiatives, including a benchmark framework and suite.
Results
The paper presents a benchmark-suite approach intended to support end users in applying ML benchmarks and developing new benchmarks.
Takeaways & Limitations
Scientific ML benchmarking can organize reusable scientific applications and provide practical support for extending benchmark coverage.
Takeaways & Limitations
Most reviewed initiatives do not address efficient distribution of large datasets and rely on externally sourced datasets.
Abstract
from arXiv · showhide
The breakthrough in Deep Learning neural networks has transformed the use of AI and machine learning technologies for the analysis of very large experimental datasets. These datasets are typically generated by large-scale experimental facilities at national laboratories. In the context of science, scientific machine learning focuses on training machines to identify patterns, trends, and anomalies to extract meaningful scientific insights from such datasets. With a new generation of experimental facilities, the rate of data generation and the scale of data volumes will increasingly require the use of more automated data analysis. At present, identifying the most appropriate machine learning algorithm for the analysis of any given scientific dataset is still a challenge for scientists. This is due to many different machine learning frameworks, computer architectures, and machine learning models. Historically, for modelling and simulation on HPC systems such problems have been addressed through benchmarking computer applications, algorithms, and architectures. Extending such a benchmarking approach and identifying metrics for the application of machine learning methods to scientific datasets is a new challenge for both scientists and computer scientists. In this paper, we describe our approach to the development of scientific machine learning benchmarks and review other approaches to benchmarking scientific machine learning.
1 Introduction
Scientific machine learning applies diverse learning approaches to increasingly large scientific datasets, but selecting suitable algorithms, data, and hardware remains difficult. The paper frames benchmarking as a way to develop evidence-based guidance and compare ML methods, architectures, and applications.
- Motivation and scope: Scientific ML uses supervised, unsupervised, and reinforcement learning to identify patterns, anomalies, trends, and other structures in scientific datasets.Supervised learning requires labelled data, whereas unsupervised learning can support clustering and anomaly identification; reinforcement learning uses rewards and penalties.
- Motivation and scope: Label scarcity makes supervised learning difficult for experimental datasets, while simulated training data may not represent real data well enough for satisfactory inference.The paper identifies this mismatch as a limitation of using simulation to obtain labels.
- Motivation and scope: Selecting an effective ML algorithm depends on training-data type, quantity, quality, labels, scientific task, required performance, and available hardware.The paper describes this as a multidimensional choice spanning algorithms, architectures, and scientific problems.
- Benchmarking approach: The paper proposes scientific ML benchmarks to develop guidelines and best practices grounded in real-world applications and relevant data.These guidelines are intended to assist scientific users and identify areas needing further research in algorithms, architectures, and software.
- Benchmarking approach: Scientific ML benchmarks act as blueprints for scientific problems, enable comparisons across the ML ecosystem, and include datasets unlike conventional HPC benchmarks.The paper emphasizes open curated datasets as an important foundation for using ML in research and directing further ML research.
- Benchmarking approach: The paper defines the benchmark scope and challenges, reviews existing initiatives, and discusses SciMLBench as a recent scientific ML benchmarking initiative.The structure moves from benchmark concepts and design considerations to initiative reviews and conclusions.
2. Machine Learning Benchmarks for Science
A scientific ML benchmark combines a scientific problem, its dataset, and a reference implementation, while benchmark suites collect such benchmarks across domains. The paper distinguishes scientific, application, and system benchmarking by their targets, fixed assets, and performance measures.
- Elements of a benchmark: A scientific ML benchmark contains a dataset and a reference implementation tied to a scientific problem; a suite is a collection of such benchmarks.The reference implementation may be written in languages such as Python or C++.
- Focus of benchmarking: Scientific ML benchmarking tests algorithmic improvements on fixed data and typically fixed hardware and software against scientific objectives.Its metrics can include F1 score, training accuracy, and time-to-solution.
- Focus of benchmarking: Application benchmarking evaluates complete ML applications across hardware and software environments, commonly using throughput, time-to-solution, and scaling.The application and its data provide the context for evaluating the overall system, including libraries, runtimes, and filesystems.
- Focus of benchmarking: System benchmarking isolates performance effects of hardware architecture on specific operations, using measures such as time-to-solution, FLOP/s, and data movement.This focus is more operation-specific than application benchmarking and is less dependent on the broader system environment.
- Examples: Scientific ML benchmarks cover applications including material-phase classification, galaxy photometric-redshift estimation, microcrack clustering, and microscope-image denoising.These examples span supervised classification, estimation, unsupervised learning, and regression across materials science and astronomy.
3 Benchmarking Process
Scientific ML benchmarking must coordinate applications, datasets, metrics, frameworks, reporting, and compliance while addressing practical challenges in data access, coverage, and extensibility.
- Framework: Benchmark suites should unify common application aspects, including portability, flexibility, and logging.
- Metrics of Choice: Metrics should match the benchmark focus: science-specific comparisons may use varying metrics, whereas system-level evaluations can share common metrics across applications.
- Framework: A coherent framework reduces the complex benchmarking operations required when users evaluate disparate applications for specific goals.
- Reporting and Compliance: Reporting and compliance requirements help ensure benchmarking is conducted fairly across different hardware platforms.
- Data: Curated, large-scale experimental or simulated datasets are essential, but their curation, maintenance, and distribution are challenging.
- Distribution: Terabyte-scale datasets make access and downloading difficult, requiring benchmarks to provide both reference implementations and relevant datasets.
- Coverage and Extensibility: Benchmark suites should cover diverse scientific disciplines and methods while remaining extensible and minimizing application code refactoring.
4. Review of Benchmarking Initiatives
Scientific ML benchmarking is a relatively new requirement compared with established ML competitions and broader benchmarking initiatives. The review compares initiatives by focus, process, and challenges, noting that support is often partial.
- Motivation: ImageNet helped drive innovation in deep learning, whereas providing applications, guidelines, and best practices for scientific ML remains relatively new.
- Review scope: Conventional benchmarks for HPC systems, compilers, memory, storage, and networking are excluded from the review.
- Review scope: The review evaluates initiatives by benchmark focus, process, and challenges: science, application, system; metrics, framework, reporting; and data, distribution, coverage, extensibility.
- Initiatives: The review covers initiatives including Deep500, RLBench, CORAL-2, DAWNBench, AI Bench, MLCommons, SciML-Bench, and community competitions.
- Scope: Individual benchmarking initiatives may support only some of the identified benchmark aspects.
4.1 Deep 500
Deep500 provides a customizable, modular infrastructure for comparing deep learning components and techniques, but adoption and benchmark coverage remain limited by user-side code and application requirements.
- Approach: Deep500 proposes customizable, modular infrastructure for comparing deep learning frameworks, algorithms, libraries, and techniques.
- Approach: Its design factorizes deep learning into operators, network processing, training, and distributed training.
- Limitations: Mapping code to a new framework has impeded adoption for new benchmark development.
- Limitations: Deep500 does not include representative applications or benchmarks by default, leaving them and reporting support to end users.
- Comparison: RLBench provides hundreds of hand-crafted tasks for evaluating developments in reinforcement learning and related areas, but lacks support for the benchmark classes discussed earlier.
4.3 CORAL-2
CORAL-2 combines broad scientific coverage with ML suites targeting computational performance, memory, and communication characteristics. Its ML technique coverage and attention to data distribution are limited.
- Benchmark focus: CORAL-2 benchmarks computational problems relevant to scientific domains or data science, emphasizing scalability rather than science alone.
- Evaluation: The benchmark codes are typically community-backed and can demonstrate the performance characteristics of proposed hardware on relevant computational workloads.
- Suites: Its ML and data-science tools include big data analytics and deep learning suites.
- Suites: The BDS suite covers PCA, k-means clustering, and SVM, while the DLS suite relies primarily on ImageNet and CANDLE benchmarks.
- Benchmark targets: BDS exercises memory constraints through PCA, computing capabilities through SVM, or both through k-means, and also addresses communication characteristics.
- Limitations: CORAL-2 has broad science coverage, but limited ML technique coverage and little focus on scientific data distribution for algorithm improvement.
4.4 AI Bench
AI Bench is a broad benchmark environment for machine-learning tasks across internet services, using system performance for fixed targets rather than a specific science focus.
- AI Bench is supported by the International Open Benchmark Council, a nonprofit organization promoting standardization and benchmarking of emerging technologies.
- Its scope spans search, social networks, and e-commerce, with tasks including image classification, generation, translation, object detection, summarization, advertising, and natural language processing.
- AI Bench uses open datasets and takes system performance for a fixed target as its primary metric.
- The environment enforces some reporting compliance but lacks a specific science focus and a fully specified framework.
4.6 Benchmarks from MLCommons Working Groups
MLCommons organizes benchmark efforts across science, application, and system concerns, with separate working groups emphasizing scientific applications and HPC-scale deep learning.
- MLCommons is an international initiative aimed at improving all aspects of the machine-learning landscape, including benchmarking, datasets, and best practices.
- HPC working group: The MLCommons HPC benchmark suite targets scientific applications using deep learning at HPC scale.
- HPC working group: Its codes and data support analysis of system performance, including model-system interactions, workload optimization, and execution bottlenecks.
- Science working group: The MLCommons Science benchmark suite focuses on applying machine learning to scientific applications across several domains.
- Science working group: The Science suite lacks a supportive execution framework but enforces reporting compliance, and its benchmarks cover science, application, and system dimensions.
4.7 SciMLBench
SciMLBench is a scientific machine-learning benchmark suite designed to cover a broad range of scientific benchmarking concerns. In contrast, community competitions lack a coherent, controlled methodology.
- SciMLBench was developed specifically for scientific machine learning and covers nearly every aspect of the benchmark cases reviewed in the paper.
- Kaggle-style competitions do not provide a coherent methodology or controlled approach for developing benchmarks.
- These competitions lack a benchmark-running framework and data-distribution methods, while each uses its own dataset, rules, and compliance metrics.
- Because competitions are typically short-lived, they are unlikely to deliver broadly applicable best practices or guidelines.
5. The SciML-Bench Approach
The SciMLBench approach combines a framework, scientific benchmarks, and associated FAIR datasets into a scalable system for science, application, and system-level evaluation. Its design emphasizes extensibility, interoperability, customization, and controlled data handling.
- SciMLBench uniquely addresses nearly all reviewed concerns while focusing specifically on scientific machine learning and offering broad versatility.
- The suite has three components: a framework, machine-learning benchmarks for scientific tasks, and associated open, domain-specific, FAIR datasets.
- Framework: The framework simplifies benchmarking, logging, reporting, and benchmark development through a coherent API, while remaining extensible, customizable, and architecture-independent.
- Benchmarks: Benchmarks range from single systems to large supercomputers and include scientific applications such as material-structure classification, cloud masking, and electron-microscopy denoising.
- Datasets: The released examples use datasets ranging from 5GB to 1.2TB, including simulated images, multispectral satellite imagery, and noisy or denoised microscopy images.
- Metrics: Detailed logging supports science, application, and system-level benchmarking by collecting performance information, including end-to-end I/O and network behavior.
- Metrics: Users choose the final reported metrics, which can support figures of merit comparing models or hardware systems for the same problem.
- Data curation and distribution: Datasets are separated from code, stored in object storage, and downloaded on demand so users avoid unnecessary transfers of large data.
6 Outlook and Conclusions
The paper argues that scientific machine learning benchmarks must address challenges in benchmark focus, processes, data distribution, and extensibility. It presents SciMLBench as a framework intended to support effective benchmark use, development, and extension.
- Scope and challenges: Scientific ML benchmarking spans challenges in benchmark focus, benchmarking processes, and developing useful benchmark suites.The paper identifies these issues across Sections 2.2, 2.3, and 3.
- Design requirements: A useful scientific ML suite must provide more than a disparate collection of applications; it must support effective use and new benchmark development.The suite should also enable users to extend it for their own purposes.
- Gaps in existing approaches: Existing benchmark initiatives often focus on scientific applications but rarely address efficient distribution of large datasets.The review found that nearly all initiatives omit this data-distribution problem.
- Gaps in existing approaches: Many approaches rely on externally sourced datasets while implicitly expecting users to handle associated data issues.This limits how much support the benchmark itself provides for data-related concerns.
- SciMLBench: SciMLBench addresses most identified concerns through a benchmark framework designed for easy extensibility.The paper reviews the initiative as a framework intended to support the scientific community in exploiting ML methods.
- Assessment framework: The paper uses qualitative traffic-light assessments to indicate whether benchmarking approaches provide none, partial, questionable, or full support for the identified concerns.Table 1 summarizes the overall assessment of the approaches.