PaperlayerView exact revision on arXiv

Source-linked AI summary

VECMAtk: A Scalable Verification, Validation and Uncertainty Quantification Toolkit for Scientific Simulations

D. Groen, H. Arabnejad, V. Jancauskas, W. N. Edeling, F. Jansson, R. A. Richardson, J. Lakhlili, L. Veen, B. Bosak, P. Kopta, D. W. Wright, N. Monnier, P. Karlshoefer, D. Suleimenova, R. Sinclair, M. Vassaux, A. Nikishova, M. Bieniek, O. O. Luk, M. Kulczewski, E. Raffin, D. Crommelin, O. Hoenen, D. P. Coster, T. Piontek, P. V. Coveney

arXiv:2010.03923v2cs.MS

TL;DR

Uncertain inputs and modelling assumptions complicate efforts to establish whether scientific simulation results are trustworthy. VECMAtk provides reusable VVUQ procedures and tools for workflows across local and HPC settings. The paper reports improvements to its components and describes toolkit capabilities, applications, and performance.

  • Problem

    Scientific models use uncertain inputs and simplifying assumptions, while their results may be approximate, making verification, validation, and uncertainty quantification relevant to assessing trust.

    Source: §1 · p. 2, (c) Climate · p. 11

  • Method

    VECMAtk combines reusable VVUQ patterns and workflow components, including EasyVVUQ and tools for validation and parallel execution.

    Source: §1 · p. 2, (a) EasyVVUQ · p. 5, (a) Fusion · p. 11

  • Results

    The toolkit combines VVUQ procedures with automated workflows and large job-ensemble execution; FabSim3 with QCG-PJ reduces submission overhead by a factor 2 for 100-job ensembles to about 0.35 seconds per job.

    Source: §7 · p. 17, §6 · p. 17

  • Takeaways & Limitations

    The toolkit aims to support reusable VVUQ procedures across application domains so simulation results can inform decision-making responsibly.

    Source: §7 · p. 18

    The development approach is shaped by requirements for minimal application changes, support for any domain, flexible recombination, and geographically distributed users and developers.

    Source: §3 · p. 4

Abstract

from arXiv · show

We present the VECMA toolkit (VECMAtk), a flexible software environment for single and multiscale simulations that introduces directly applicable and reusable procedures for verification, validation (V&V), sensitivity analysis (SA) and uncertainty quantification (UQ). It enables users to verify key aspects of their applications, systematically compare and validate the simulation outputs against observational or benchmark data, and run simulations conveniently on any platform from the desktop to current multi-petascale computers. In this sequel to our paper on VECMAtk which we presented last year, we focus on a range of functional and performance improvements that we have introduced, cover newly introduced components, and applications examples from seven different domains such as conflict modelling and environmental sciences. We also present several implemented patterns for UQ/SA and V&V, and guide the reader through one example concerning COVID-19 modelling in detail.

1. Introduction

Scientific simulations can diverge from reality because inputs are uncertain and models rely on simplifying assumptions. The paper presents VECMAtk as a toolkit for applying VVUQ procedures to assess model accuracy and trustworthiness.

1. Introduction

  • Uncertain inputs and simplifying assumptions can cause computational models to behave differently from real-world systems, making accuracy checks across realistic scenarios necessary.Inputs may include initial and boundary conditions and model parameters that are unknown or inadequately described.Source: §1 · p. 2
  • VVUQ combines verification of computational models against their mathematical models, validation against experimental data, and quantification of how parameter variations affect outcomes.These processes provide a basis for assessing trust in models and their results [59].Source: §1 · p. 2
  • For complex, multiscale, or multiphysics simulations, VVUQ helps determine whether approximate results can be trusted and used by decision makers.The paper links simulation impact to trustworthiness.Source: §1 · p. 2
  • VECMAtk provides reusable VVUQ techniques and Verification and Validation Patterns for comparing simulation results with benchmarks or measurements.The paper describes the latest toolkit release and introduces its components and application examples.Source: §1 · p. 2

2. Related Work

Related work spans toolkits for VVUQ, HPC ensemble execution, surrogate modelling, and statistical analysis. The section positions VECMAtk among tools that cover subsets of these capabilities.

2. Related Work

  • DAKOTA offers optimization, UQ, parameter-study, and calibration algorithms, but has a steep learning curve and no coordination for resources across concurrent runs.UQ-focused alternatives include UQTK and UQLab, while multi-level Monte Carlo is identified as an efficient UQ method.Source: §2 · p. 3
  • HPC-related tools cover probabilistic modelling, ensemble execution, surrogate modelling, and data-driven workflows, including OpenTURNS [10], RADICAL-Cybertools [11], GPM/SA, Swift/T, and Parsl.Uranie supports UQ and SA on local and HPC resources.Source: §2 · p. 3
  • Statistical tools provide UQ, SA, calibration, uncertain-data generation, or Bayesian modelling through methods including QMC, PCE, Sobol’s method, and MCMC.Examples include Uncertainpy, PSUADE, DUE, PyMC3, SimLab, UQLab, and SAFE.Source: §2 · p. 3

3. The VECMA toolkit (VECMAtk)

VECMAtk is designed as a flexible collection of reusable VVUQ patterns and components for application-specific workflows. Its development and release approach also emphasizes broad applicability and continuing component improvements.

3. The VECMA toolkit (VECMAtk)

  • VECMAtk implements UQ and sensitivity-analysis patterns and verification and validation patterns, supporting execution on local machines and petascale and emerging exascale resources.The toolkit aims to make VVUQ patterns reusable across application domains and workflows.Source: §3 · pp. 3–4
  • The development approach is shaped by minimal application changes, support for any domain, flexible recombination, and geographically distributed users and developers.These four factors guide the toolkit’s design.Source: §3 · p. 4
  • VECMAtk elements can be recombined through stable interfaces, data formats, and APIs to build application-specific VVUQ workflows across scales and scale bridges.The applications map to Extreme Scaling, Replica Computing, and Heterogeneous Multiscale Computing patterns.Source: §3 · p. 4

(a) Release schedule and changes compared to the initial release

  • Since the June 2019 first major release, all components have gained scalability, features, revised tutorials, improved documentation, and user-requested bug fixes.Minor releases are scheduled every three months, while major releases include extensive documentation, tutorials, examples, training, and uptake activities.Source: (a) Release schedule and changes… · p. 4

4. Overview of the VECMAtk components

VECMAtk combines four components that can be flexibly assembled into application workflows. EasyVVUQ coordinates key stages of non-intrusive VVUQ workflows while leaving simulation execution to external tools or users.

4. Overview of the VECMAtk components

  • EasyVVUQ, FabSim3, MUSCLE 3, and QCG tools respectively support VVUQ workflows, computational research automation, multiscale coupling, and advanced HPC workflow design and execution.Components can be combined with each other or third-party tools, and no single component is essential to every application.Source: §4 · p. 4

(a) EasyVVUQ

  • EasyVVUQ coordinates sample generation through execution and analysis for non-intrusive VVUQ, while execution itself is external and can use middleware or be performed manually.It tracks and logs sampling steps to support restartability and failure resistance, including workflows with thousands to millions of runs.Source: (a) EasyVVUQ · p. 5
  • EasyVVUQ’s five stages are application description, sampling, external execution, collation of outputs, and statistical analysis.Application description includes producing simulation inputs, parsing outputs, and specifying how the simulation runs.Source: (a) EasyVVUQ · p. 5
  • EasyVVUQ supports variance-driven sensitivity analysis, stochastic collocation, polynomial chaos expansion, and basic statistical methods such as bootstrapping.Its components are extensible, while built-in functionality often requires users only to define input and output formats and perform analysis.Source: (a) EasyVVUQ · p. 5

(b) FabSim3

FabSim3 automates complex computational workflows and supports customization across applications and machines. In VECMAtk, it connects application-specific information with VVUQ patterns and can submit ensemble runs through pilot jobs.

(b) FabSim3

  • FabSim3 automates creating, managing, executing, and modifying complex workflows, while curating environment and state variables to support testing and debugging.It is designed to be customized for new machines and simulation codes.Source: (b) FabSim3 · p. 6
  • FabSim3 provides adaptable code-coupling mechanisms and domain-agnostic patterns as building blocks for VVUQ in new applications.Figure 2 gives a high-level overview of its architecture, including selected plugins, patterns, and back ends; green boxes are complete and yellow boxes are still being extended.Source: (b) FabSim3 · p. 6, Figure 2 · p. 6
  • In VECMAtk, FabSim3 introduces application-specific information, combines UQPs and VVPs, and curates large sets of production runs.It can submit ensemble runs through QCG-Client or QCG-SDK pilot jobs, which manage sub-jobs without each independently waiting for resources.Source: (b) FabSim3 · p. 6

(c) MUSCLE 3

MUSCLE 3 couples scale-separated submodels into distributed simulations, with configuration-based communication and support for multiscale UQ workflows.

(c) MUSCLE 3

  • MUSCLE 3 starts scale-separated submodels in parallel and routes their network messages using a configuration file, so components need not know about one another.The MMSF may help determine how submodels should be coupled given their relative spatial and temporal scales.Source: (c) MUSCLE 3 · p. 6
  • For time-scale-separated macro–micro coupling, MUSCLE 3 can rerun the micro model as needed to match the macro model’s time steps.It also supports space-scale separation by coupling one macro model to multiple micro-model instances through vector ports.Source: (c) MUSCLE 3 · pp. 6–7
  • MUSCLE 3 supports UQ ensembles through parameter overlays, vector ports, and multiple submodel instances, while submodels can remain unaltered.Semi-intrusive UQ patterns can be added by changing the configuration and connections.Source: (c) MUSCLE 3 · p. 7

(d) QCG Tools

The QCG tools support HPC workflow design and execution, from remote job submission and flexible pilot-job scheduling to EasyVVUQ integration and graphical monitoring.

(i) QCG-Client

  • QCG-Client submits workflows, parameter sweeps, and array jobs to one or multiple clusters, using QCG-Broker to manage workflow execution and resource selection.It can be deployed as a container and integrated with FabSim3.Source: (i) QCG-Client · p. 7

(ii) QCG Pilot Job (QCG-PJ)

  • QCG-PJ is VECMAtk’s default tool for the large, dynamically created simulation ensembles needed by UQ, in part because users can install it without administrator rights.It can also be set up without other QCG components.Source: (ii) QCG Pilot Job (QCG-PJ) · p. 7
  • QCG-PJ runs many small jobs within one HPC allocation, avoiding independent scheduling queues and accommodating varied resource requirements in multiscale workflows.Its manager handles workflow sub-jobs and dependencies; file-based interaction suits static workloads, while a network interface supports dynamic requests and run-time tracking. A local mode supports testing without a scheduling allocation.Source: (ii) QCG Pilot Job (QCG-PJ) · p. 7

(iii) EQI

  • EQI provides EasyVVUQ workflows with a domain-oriented API for QCG-PJ, including predefined tasks for execution and encoding on HPC resources.Its development cycle is synchronized with EasyVVUQ and QCG-PJ releases.Source: (iii) EQI · p. 8

(iv) QCG-Now

  • QCG-Now adds a graphical option for submitting and managing HPC jobs, with notifications, data transfer, monitoring, and reusable execution templates.Its embedded QCG-Monitoring view tracks long-running EasyVVUQ and QCG-PJ workflows.Source: (iv) QCG-Now · p. 8, Figure 3 · p. 8

(e) VECMAtk workflows

VECMAtk components can be combined into application workflows, including integrations with third-party tools. The examples illustrate different component combinations and user entry points.

(e) VECMAtk workflows

  • Figure 4 maps VECMAtk components to exemplar applications, illustrating component combinations while maintaining a limited deployment footprint.The components can also be integrated with third-party components.Source: Figure 4 · p. 9, (e) VECMAtk workflows · p. 9
  • Common combinations include FabSim3 with QCG-Client for remote submission, FabSim3 with EasyVVUQ for ensemble workflows, and EasyVVUQ with QCG-PJ and EQI for pilot-job execution.FabSim3 also supports QCG-PJ pilot-job creation and management.Source: (e) VECMAtk workflows · p. 9
  • Applications can replace EasyVVUQ with a MOGP emulator or use Dask instead of QCG-PilotJob for ensemble submission; other execution engines may suit particular applications or resources.Examples of other tools include RADICAL-Cybertools, Swift/T, and Parsl.Source: (e) VECMAtk workflows · pp. 9–10
  • The toolkit offers different entry points: EasyVVUQ for VVUQ, FabSim3 for curated workflows, QCG-Now for graphical management, and MUSCLE3 for multiscale coupling.Source: (e) VECMAtk workflows · p. 10

5. Exemplar applications

The applications span seven scientific domains and demonstrate VECMAtk workflows for UQ, sensitivity analysis, and—in two cases—systematic validation. The examples range from coupled simulation workflows to a COVID-19 sensitivity analysis showing that influential parameters change over time.

(a) Fusion

  • For fusion, EasyVVUQ and QCG-PJ run parallel sample batches, while the ValidationSimilarity VVP compares simulation and experimental quantity-of-interest distributions using distance metrics.Available metrics include Hellinger, Jensen–Shannon, and Wasserstein distances.Source: (a) Fusion · pp. 10–11

(b) Forced human migration

  • The forced-migration workflow automates refugee-movement ensembles, computes parameter sensitivities, couples models, and validates outputs against conflict data using EnsembleValidation.Its advanced workflows have involved up to 16,384 jobs.Source: (b) Forced human migration · p. 11

(c) Climate

  • Climate workflows use EasyVVUQ to quantify uncertainty in time-averaged statistics and FabSim3 to run ensembles on HPC resources, considering uncertain inputs, model choices, numerical settings, and turbulence.The work also investigates data-driven stochastic surrogates for model-form uncertainty.Source: (c) Climate · pp. 11–12, Figure 7 · p. 12

(d) Advanced materials

  • For advanced materials, VECMAtk automates ensemble generation, execution, collation, and analysis to gather statistics and explore input parameters with minimal human oversight.The application addresses a field where development can take over 20 years and engineering requires understanding across multiple scales.Source: (d) Advanced materials · p. 13

(e) Urban air pollution

  • UrbanAir uses VECMAtk to automate sampling, HPC ensemble execution, collation, and analysis, then applies sensitivity analysis to identify important inputs and limit future ensembles.The multiscale prediction addresses uncertainty from an incomplete emissions database.Source: (e) Urban air pollution · p. 13

(f) Biomedicine

  • The biomedical example reports prior ISR2D uncertainty and sensitivity analyses, while ISR3D UQ/SA design remains ongoing and plans on-the-fly metamodel training with cross-validation.The planned test concerns how approximation error affects micro-model results.Source: (f) Biomedicine · pp. 13–14

(g) Coronavirus modelling

  • As the COVID-19 simulation progresses, deaths are mainly sensitive to infection rate and mild recovery period, while during the first 20 days all but non-mild recovery period have significant effects.The first-order Sobol indices measure the fraction of output variance attributable to a single input parameter.Source: (g) Coronavirus modelling · p. 15, Figure 9 · p. 15

6. Toolkit performance and scalability

VECMAtk addresses ensemble sampling, job management, data transfer, and coupled-model data movement as performance bottlenecks. Its reported submission tests show lower per-job overhead with QCG-PJ for larger ensembles, while other components support scalable execution and reduce data staging.

Sampling and input creation

  • EasyVVUQ’s SQL backend handles tens of thousands of samples, tracks job status for restarting failures, and permits staged sampling without losing stored results.Ensemble submission can be delegated to scalable components such as Dask or QCG-PJ.Source: §6 · p. 16

Ensemble execution

  • QCG-PJ has been shown to execute 10,000 jobs with less than 10% overhead, including jobs lasting only one second.The paper notes that ensembles of 1,000 or more runs cannot be executed as individual jobs on most supercomputers due to scheduler constraints.Source: §6 · p. 16

Submission performance

  • With FabSim3 and QCG-PJ, job-submission overhead is about 0.35 seconds per job for 100-job ensembles and about 0.275 seconds per job for 1,000-job ensembles, a factor-of-2 reduction at 100 jobs.Without QCG-PJ, single-thread overhead is 2–3 seconds per job and falls below one second with four or more threads; Figure 10 reports the submission-time test.Source: §6 · pp. 16–17, Figure 10 · p. 16

Data movement

  • FabSim3 and EasyVVUQ automate data curation, encoding, decoding, and file staging, while EQI brings ensemble computations to HPC resources where data resides.The integrated workflow can run iteratively and dynamically in one Python script, with input files staged once in advance.Source: §6 · p. 17

Coupled-model data movement

  • MUSCLE 3 addresses data movement between coupled models and can eliminate the need for file I/O during coupling.Source: §6 · p. 17

7. Conclusions

The paper concludes that VECMAtk combines reusable VVUQ procedures with automated execution of large ensembles across diverse scientific applications. Its flexible components and open examples support reuse across domains, with the stated aim of helping make simulation results actionable for decision-making.

Conclusion

  • VECMAtk combines a range of VVUQ procedures with streamlined automation and accessible execution of large job ensembles on pre-exascale resources.Its components can be flexibly combined while retaining a limited deployment footprint.Source: §7 · p. 17
  • The examples cover fusion, forced migration, climate, advanced materials, urban air pollution, biomedical simulation, and coronavirus modelling, and are open source with tutorials for reuse as building blocks.Source: §7 · pp. 17–18
  • The toolkit aims to make VVUQ procedures and algorithms reusable across application domains so uncertainties, sensitivities, and errors can be quantified in simulations used to inform decisions.Source: §7 · p. 18

Appendix A: improvements since first VECMAtk release

Appendix A presents the fundamental component changes made since VECMAtk’s first major release, identified as June 2019.

Appendix A

Loading 2010.03923v2…