Source-linked AI summary
ACEMD: Accelerating bio-molecular dynamics in the microsecond time-scale
M. J. Harvey, G. Giupponi, G. De Fabritiis
TL;DR
Molecular dynamics needs greater computational reach to routinely explore biologically relevant timescales. The paper presents ACEMD, a production-class MD program optimized for Nvidia GPUs and running the complete computation on the GPU. ACEMD reaches microsecond-scale simulations on cost-effective workstation hardware, including a three-GPU workstation.
Problem
Routine molecular dynamics exploration of biological timescales remains computationally difficult.
Method
ACEMD is a production-class MD program optimized for Nvidia GPUs that runs the whole computation on the GPU and implements typical MD features.
Results
ACEMD reaches microsecond-scale simulations on cost-effective workstation hardware, with simulations extending to the microsecond scale over a few weeks on suitable GPU-equipped machines.
Takeaways & Limitations
Workstation computers equipped with one or more GPUs can support simulations lasting a few weeks and reach microsecond timescales.
Abstract
from arXiv · showhide
The high arithmetic performance and intrinsic parallelism of recent graphical processing units (GPUs) can offer a technological edge for molecular dynamics simulations. ACEMD is a production-class bio-molecular dynamics (MD) simulation program designed specifically for GPUs which is able to achieve supercomputing scale performance of 40 nanoseconds/day for all-atom protein systems with over 23,000 atoms. We illustrate the characteristics of the code, its validation and performance. We also run a microsecond-long trajectory for an all-atom molecular system in explicit TIP3P water on a single workstation computer equipped with just 3 GPUs. This performance on cost effective hardware allows ACEMD to reach microsecond timescales routinely with important implications in terms of scientific applications.
I. INTRODUCTION
Molecular dynamics aims to reach biologically relevant microsecond-to-millisecond timescales, but routine exploration remains computationally difficult. ACEMD addresses this challenge with a production-class GPU implementation that runs the full simulation and supports typical MD features.
- Microsecond-to-millisecond macromolecular simulations remain beyond routine computational capacity, limiting exploration of biological timescales.
- Dedicated highly parallel machines face interconnection latency and bandwidth limits, while their cost and allocation constraints restrict long simulations.
- ACEMD is optimized for Nvidia GPUs with the aim of routinely performing microsecond-scale molecular simulations.
- ACEMD maximizes performance by running the whole computation on the GPU rather than offloading only selected expensive components.
- The package implements typical production-simulation features, including PME electrostatics, thermostatic control, bond constraints, force fields, and scripting for complex protocols.
II. GPU ARCHITECTURE
Nvidia GPUs expose highly parallel, data-parallel execution through multiprocessors, threads, blocks, shared memory, and CUDA. Their architecture offers high single-precision throughput but requires careful handling of memory latency and does not transparently coordinate multiple devices.
- CUDA abstracts GPU implementation details and provides portable C-like programming for data-parallel computation.
- Nvidia GPUs use multiprocessors that synchronously execute 32 threads in parallel and manage up to 1024 concurrent threads.
- Current Nvidia devices reach up to 933 Gflops in single precision, compared with a contemporary quad-core Intel Xeon CPU.
- Each multiprocessor provides registers and low-latency shared memory, while uncached global-memory access can cost up to 400 cycles.
- Texture-unit reads can cache selected global-memory access patterns and perform linear interpolation on multidimensional floating-point arrays.
- GPU kernels execute as blocks of concurrent threads scheduled across available hardware by the CUDA runtime.
- The CUDA model does not transparently provide parallel computation across multiple GPU devices.
III. MOLECULAR DYNAMICS ON THE GPU
ACEMD implements the components of molecular dynamics on CUDA-compatible GPUs, using GPU-oriented algorithms for force evaluation, constraints, electrostatics, and supported simulation workflows.
- ACEMD implements all features of an MD simulation on a CUDA-compatible GPU.
- ACEMD supports PME, Langevin thermostating, hydrogen-bond constraints, CHARMM27 and Amber99 force fields, and common MD file formats.
- NPT simulations are not possible because the current implementation does not contain a barostat.
- Non-bonded forces dominate MD computational cost, motivating ACEMD’s cell-list algorithm for binning particles by coordinates.
- A 12 Å cutoff with R/2 bins yields approximately 22 atoms per cell, comparable to the 32-thread warp size on current Nvidia GPUs.
- ACEMD uses a maximum cell population of 64 to accommodate transient density fluctuations that can exceed the GPU warp size.
- The GPU force kernel loads neighboring cells into shared memory and computes Lennard-Jones and electrostatic forces for particles in each cell.
IV. SINGLE-PRECISION FLOATING-POINT ARITHMETIC VALIDATION
The validation tests whether ACEMD’s single-precision GPU implementation preserves energy and structural observables across integration settings and electrostatics methods.
- ACEMD uses single-floating-point arithmetic because GPUs deliver much higher performance in single precision than double precision.
- Energy-conservation tests used DHFR systems with 1, 2, and 4 fs timesteps, rigid constraints, harmonic bonds, and hydrogen mass repartitioning.
- Energy conservation remained reasonably good with larger timesteps and combined M-shake and hydrogen mass repartitioning.
- Hydrogen mass repartitioning increases the timestep to 4 fs while affecting only dynamic properties, not the equilibrium distribution.
- ACEMD’s Na-Na pair distribution functions agree well with the Gromacs reference for all tested integration timesteps.
- PME consistently gives better results than a 12 Å cutoff for the tested homogeneous NaCl-water system.
V. PERFORMANCE
ACEMD distributes work across multiple GPUs and dynamically load-balances heterogeneous systems. On a DHFR benchmark, three GPUs achieved performance comparable to large CPU configurations.
- ACEMD dynamically load-balances computation across GPUs, accommodating heterogeneous molecular systems and host architectures.
- 17.55 ms per step with the DHFR system decreased to 7.56 ms per step when parallelized across three GPUs.
- ACEMD achieves a parallel efficiency of 2.3 over three GPUs using its task-decomposition scheme.
- Further device-to-device communication could improve performance by enabling spatial-decomposition strategies such as neutral-territory schemes.
- ACEMD’s DHFR performance approached that of 256 CPU cores using NAMD and 64 CPU cores using Desmond on a fast-interconnect cluster.
- More than 45 ns/day was achieved with three GPUs and almost 20 ns/day with one GPU using 4 fs integration and hydrogen mass repartitioning.
- Unlike enhanced NAMD, ACEMD is designed for optimal performance on a small number of GPUs rather than scaling across multiple GPU-equipped hosts.
VI. MICROSECOND SIMULATIONS ON WORKSTATION HARDWARE
ACEMD enabled a microsecond-scale Villin simulation on a workstation with three GPUs, using an unfolded starting state and explicit TIP3P water. The trajectory repeatedly sampled the crystal-like overall shape but did not converge to it, while performance reached 66 nanoseconds/day.
- Microsecond trajectory: A single workstation with 3 Nvidia GPUs produced a microsecond-long trajectory for the 13,701-atom Villin system in TIP3P water.The production run used 720 GPU cores and required approximately 15 days.
- Trajectory analysis: 4.87 Å was the minimum backbone RMSD observed during the microsecond trajectory.The best structure at this RMSD was overlapped with the crystal structure.
- Trajectory analysis: The protein often sampled the crystal structure’s overall shape but did not converge toward it within the simulation window.The simulation began from an unfolded configuration after a 40-nanosecond run at 450 K.
- Performance: 66 nanoseconds/day was achieved for the Villin production run on the 3-GPU workstation.The authors describe this as probably representing the limit of the then-current hardware and software implementation.
- Kernel performance: A 66 Å cubic simulation box with 6 Å cells could scale over 167 multiprocessors, corresponding to 6 G200-class GPUs.The stated scaling limit reflects simultaneous processing of available blocks by the multiprocessors.
- Kernel performance: The non-bonded kernels had a minimum fully parallel runtime of 3.4 ms/step on current hardware.The analysis examined inner and outer kernels as a function of block count per invocation.
VII. CONCLUSIONS
The authors present ACEMD as a GPU-based molecular dynamics application that reaches microsecond timescales on cost-effective workstation hardware. They report broad deployment and performance while identifying limits in atom count, GPU parallelization, and cluster scaling.
- Conclusions: ACEMD was designed to reach the microsecond timescale using computationally cost-effective workstation hardware.The authors frame this as the central capability demonstrated by the work.
- Practical implications: Tens of nanoseconds per day make simulations up to the microsecond scale feasible within a few weeks on suitable GPU-equipped workstations.The authors characterize such runtimes as reasonable for workstation-class computers with one or more GPUs.
- Limitations and future work: The current implementation limits system size to 250,000 atoms, although this limit could be extended.The paper also anticipates that cluster scaling would require spatial decomposition rather than the current task-parallel scheme.
- Validation and deployment: ACEMD was extensively tested through deployment on several thousand GPU-equipped PCs in the GPUGRID.net volunteer distributed-computing project.The project uses BOINC middleware.
- Validation and deployment: Over 30 Tflops of sustained performance and thousands of nanosecond-long trajectories per day were reported for GPUGRID.net.The infrastructure supported high-throughput molecular simulations, including accurate virtual screening.
- Limitations and future work: The current implementation limits parallel performance to 3 GPUs because it uses simple task parallelization.The authors planned extensions targeting scalability across 1–32 GPUs.
- Limitations and future work: ACEMD executes entirely on GPU devices with CPU loads within 5%, potentially supporting strong scaling on GPU-attached machines.The authors state that efficient GPU-cluster scaling would require refactoring the parallelization scheme.