Source-linked AI summary
Building a Framework for Predictive Science
Michael M. McKerns, Leif Strand, Tim Sullivan, Alta Fang, Michael A. G. Aivazis
TL;DR
Expensive predictive calculations require thousands of evaluations over high-dimensional, constrained models, motivating infrastructure for distributed heterogeneous computation. The paper presents integrated optimization and computing frameworks that formulate rigorous uncertainty quantification as global optimization. The framework provides parallel execution, sensitivity analysis, probability-of-failure bounds, and model-versus-reality uncertainty measures.
Problem
Realistic predictive models may require hundreds or thousands of costly evaluations across high-dimensional, constrained, heterogeneous computing resources.
Method
The paper integrates mystic optimization with pathos, pyre, and hydra, and formulates rigorous UQ and OUQ calculations as global optimization problems.
Results
The framework provides probability-of-failure bounds, sensitivity measures, critical-region discovery, and conservative uncertainty estimates combining model variability with modeling error.
Takeaways & Limitations
Predictive-science questions can be addressed quantitatively through optimization and framework services designed for massively parallel and heterogeneous computation.
Abstract
from arXiv · showhide
Key questions that scientists and engineers typically want to address can be formulated in terms of predictive science. Questions such as: "How well does my computational model represent reality?", "What are the most important parameters in the problem?", and "What is the best next experiment to perform?" are fundamental in solving scientific problems. Mystic is a framework for massively-parallel optimization and rigorous sensitivity analysis that enables these motivating questions to be addressed quantitatively as global optimization problems. Often realistic physics, engineering, and materials models may have hundreds of input parameters, hundreds of constraints, and may require execution times of seconds or longer. In more extreme cases, realistic models may be multi-scale, and require the use of high-performance computing clusters for their evaluation. Predictive calculations, formulated as a global optimization over a potential surface in design parameter space, may require an already prohibitively large simulation to be performed hundreds, if not thousands, of times. The need to prepare, schedule, and monitor thousands of model evaluations, and dynamically explore and analyze results, is a challenging problem that requires a software infrastructure capable of distributing and managing computations on large-scale heterogeneous resources. In this paper, we present the design behind an optimization framework, and also a framework for heterogeneous computing, that when utilized together, can make computationally intractable sensitivity and optimization problems much more tractable.
Introduction
The paper motivates predictive science as global optimization and presents integrated software for massively parallel, heterogeneous computation and rigorous uncertainty quantification. Its framework addresses expensive, high-dimensional models by combining optimization, distributed execution, and uncertainty-analysis services.
- Motivation: Thousands of expensive model evaluations make global optimization impractical with standard sequential algorithms.A one-minute model evaluated 10,000 times would require approximately seven days.
- Motivation: Massively parallel optimization requires rethinking algorithms and integrating heterogeneous computing resources.The stated goal is to lower barriers to these capabilities for scientists and engineers across fields.
- Framework design: Mystic and pathos are modified to provide parallel optimization and to stage and launch optimizations across heterogeneous resources.The framework leverages pyre and hydra for integration, distributed communication, and computation management.
- Rigorous uncertainty quantification: System uncertainty can combine model-predicted variability with modeling-error uncertainty through the conservative bound U = DF + DG−F.The model diameter DF measures predicted uncertainty, while DG−F measures discrepancy between model and experiment.
- Rigorous uncertainty quantification: OUQ formulates uncertainty bounds as extreme values of optimization problems over feasible scenarios, while allowing assumptions and information to be incorporated explicitly.The framework is presented as avoiding inappropriate implicit assumptions while producing rigorous optimal bounds.
A Highly-Configurable Optimization Framework
Mystic combines configurable optimization algorithms, termination conditions, monitoring, and analysis tools behind a common solver interface.
- Mystic offers global, local, and gradient-based solvers that share an API for configuring and launching optimization jobs.Solvers can be swapped without requiring users to write new code.
- Customizable termination conditions let users configure convergence behavior for optimization problems with non-standard profiles.Termination settings can be modified and optimization continued with updated tolerances.
- Progress monitors track fitted parameters and cost values, with optional recording of gradients or other metrics.Generation and evaluation monitors can be attached separately.
- The toolkit includes conceptual optimizer components, wrappers, factories, and analysis viewers for constructing and inspecting optimization workflows.Monitoring can also report the best energy at configurable generation intervals.
Constraints Toolkit
Mystic supports generic bounds and parameter constraints, including symbolic nonlinear constraints, while separating constraint solving from cost evaluation.
- Bounds and user-defined parameter constraints can be applied generically to unconstrained optimization algorithms.Strict ranges and constraint functions are configured independently through solver methods.
- Set-based constraints transform candidates into valid points before optimization, whereas penalty methods modify cost after violations.Penalty methods apply an energy change ∆E = k · p(x) when constraints are violated.
- A SymPy-based parser translates symbolic hard or soft equality and inequality constraints into executable constraint objects.The parser accepts multiple and nonlinear constraints.
- Constraint evaluation is decoupled from cost evaluation, decomposing highly constrained optimization into K independent constraint solves followed by unconstrained optimization.This approach has been shown effective when K ≈200.
Seamless Migration to Parallel Computing
Mystic exposes parallel and distributed execution through services that wrap user models, allowing optimization components to run across heterogeneous resources.
- Mystic decomposes optimization algorithms into components that can operate across process space as interacting services.User models or cost functions are commonly mapped over parallel resources.
- Mystic includes stock models and model factories for testing, including a Lorentzian model constructed from coefficients.Factory methods insert pathos infrastructure into models.
- The modelFactory method casts user functions as callable services with pathos infrastructure and optional monitoring.A map function can evaluate the resulting service over a sequence of inputs.
- A costFactory binds a model or model service to experimental data G using a difference metric such as |F(x)−G|2.The factory accepts raw models, model proxies, or model services.
A Framework for Heterogeneous Computing
pathos provides a consistent interface for developing, launching, monitoring, and controlling parallel computations across heterogeneous resources. Its service-based architecture combines launchers, monitors, and workload-distribution strategies.
- pathos supports local development and remote execution across environments ranging from multicore workstations to large computer-cluster networks.It also provides access to job history.
- pathos includes launchers for queue-less MPI, SSH-based execution, and multiprocessing, each encoding the syntax for its execution environment.The framework also supplies map-reduce algorithms for its available launchers.
- The framework is divided into dill, pox, pyina, and pathos subpackages for serialization, filesystem utilities, MPI mapping, and distributed map-reduce.
- pathos uses pyre components in which services, launchers, monitors, and strategies abstract execution, logging, and workload distribution.Strategies may be static, such as equalportion and carddealer, or dynamic worker-pool approaches.
Globally Unique Message Passing
The framework enables distributed objects and model services to communicate and execute across remote resources. Its interfaces can construct SSH-based services and parallel maps over local and remote workers.
- hydra gives Python objects network addresses and supports asynchronous message exchange through proxy objects.Proxy calls package function names and arguments into datagrams sent to remote objects.
- A model can be exposed as an SSH-based remote service and evaluated through its proxy.
- Figure 6 depicts a distributed parallel map copying a service n times across N machines.When the mapped object is not a service, jobs are launched distributively but managed at the machine level.
- A parallel map can combine two local workers with all available IPC servers at a remote host.The resulting map evaluates the user’s model over a supplied range.
Serialization
The framework combines Python serialization and filesystem utilities with remote service management. These components support moving Python state, discovering remote environments, and securing distributed communication through SSH tunnels.
- dill extends pickle to serialize and deserialize most built-in Python and NumPy types.Serialization converts objects to byte streams and reverses that conversion.
- dill can save an interpreter session so it can be transferred to another computer and resumed later.
- pox provides filesystem and operating-system utilities that help users interact with remote hosts as if logged in locally.Its operations include discovering users, shells, filesystem roots, packages, source locations, and versions.
- pathos configures and launches IPC services remotely, supports distributed map-reduce across local and remote resources, and offers basic automated load balancing.
- Because IPC communication is insecure, pathos can wrap distributed services or communication in SSH tunnels.The paper reports distributed calculations using SSH tunneling across national-laboratory clusters and a laptop.
Parallel Staging and Launching
pyina lowers the barrier to MPI-based high-performance computing through parallel map-reduce strategies and common launcher interfaces. The carddealer-DE optimizer stages model copies in parallel across distributed resources.
- pyina lets users extend code to MPI-based high-performance computing with minimal refactoring.
- Its central parallel map-reduce element offers two strategies for distributing job lists across available nodes.Map objects hide MPI internals and allow parallel batch execution using standard Python.
- pyina provides a common interface for launching MPI jobs directly, through mpirun or mpiexec, or via schedulers such as torque and slurm.
- A custom map can execute MPI locally using four workers and the carddealer strategy.
- Figure 7 shows a carddealer-DE optimizer staging n copies of the user’s model F(x) in parallel across distributed resources.
New Massively-Parallel Optimization Algorithms
Mystic extends optimization to massively parallel execution, supporting both parallel evaluations within a solver and parallel launches of independently initialized optimizers.
- Parallel solver execution: Parallel evaluation is enabled by supplying a map to the solver, allowing each generation’s 20 candidates to execute across 4 MPI workers.Differential Evolution uses SetEvaluationMap with an MpirunCarddealer(4) configuration.
- Parallel solver execution: Nested parallelism launches N optimizers with different initial conditions, using SetNestedSolver to schedule independent searches.BuckshotSolver schedules N = 20 optimizers on 5 nodes with 4 processors each.
- Initialization strategies: A lattice-Powell optimizer starts parallel Powell searches at lattice-cuboid centers, whereas buckshot-Powell uses uniformly random initial values.Both strategies distribute local searches across multiple starting points in parameter space.
- Solver configuration: The solver examples configure model bounds, termination conditions, evaluation limits, monitors, and parallel maps through composable interfaces.The nested Powell configuration includes strict ranges and an evaluation limit of 50.
Probability and Uncertainty Tooklit
Mystic provides optimization-based services for rigorous uncertainty quantification, sensitivity analysis, and discovery of critical regions in parameter space.
- UQ services: Mystic supports probability-of-failure bounds, model-based sensitivity, critical-region discovery, and optimization over probability measures as UQ framework services.Diameter calculations can be coupled with partitioning algorithms to identify critical behavior.
- Optimization metrics: Standard model-fitting and sensitivity calculations use squared difference metrics between model predictions and data or between predictions at two parameter sets.The two metrics are |F(x)−G|2 and |F(x)−F(y)|2.
- Sensitivity analysis: Suboscillations are computed through nested optimization, with global optimizers evaluating component-wise differences before forming the diameter.The diameter is the root-mean square of its component-wise suboscillations.
- Optimal uncertainty quantification: OUQ maximizes a failure probability over probability distributions subject to information-set constraints such as bounds on expected model response.The optimized object is a product measure rather than a standard difference metric.
- Optimal uncertainty quantification: Discrete measures represent support points with weights and provide operations for mass, range, mean, and product-measure construction.OUQ optimizations use product_measure objects, strict ranges, and parameter constraints to define the feasible set.
Future Developments
Future work targets broader critical-region discovery, stronger heterogeneous-resource services, expanded analysis methods, and migration of development features into releases.
- Release status: Features presented in the paper were not all in released code, making migration from development branches a primary next step.The authors identify this migration as being of primary importance.
- Analysis and optimization extensions: Future work also includes parameter-correlation methods, treatment of data uncertainty, additional statistical and measure methods, and more parallel optimization algorithms.These extensions broaden the framework’s analysis and optimization capabilities.
- Critical-region discovery: Planned partitioning algorithms will discover critical regions using criteria such as competing subdiameters accounting for 90% or more of total diameter.The current rule instead iteratively tightens piecewise-McDiarmid-type upper bounds.
- Heterogeneous computing: Pathos services are slated for greater robustness across platforms and objects, unified map interfaces, load balancing, and expanded monitoring and logging.The planned services cover parallel and distributed resources as well as nested analysis circuits.
Summary
The paper presents a brief overview of the mathematical and software components used to build a software framework for predictive science.
- Summary: The paper surveys mathematical and software components for constructing a predictive-science software framework.Its scope is presented as a brief overview.