PaperlayerView exact revision on arXiv

Source-linked AI summary

Strong scaling of general-purpose molecular dynamics simulations on GPUs

Jens Glaser, Trung Dac Nguyen, Joshua A. Anderson, Pak Lui, Filippo Spiga, Jaime A. Millan, David C. Morse, Sharon C. Glotzer

arXiv:1412.3387v1physics.comp-ph

TL;DR

The paper addresses how to scale general-purpose GPU molecular dynamics beyond single-GPU execution despite communication latency and diminishing work per GPU. It introduces MPI spatial domain decomposition and GPU-centric communication optimizations, demonstrating equivalent or superior scaling on thousands of GPUs. The current release remains limited in supported multi-GPU force fields and workload distributions.

  • Problem

    Multi-GPU molecular dynamics must overcome device-communication latency and declining per-GPU workload to maintain strong-scaling efficiency.

  • Method

    HOOMD-blue 1.0 combines MPI spatial domain decomposition with GPU-resident communication, CUDA-aware MPI, autotuned kernels, and strong-scaling optimizations.

  • Results

    HOOMD-blue achieves equivalent or superior scaling to similarly optimized GPU-enabled molecular dynamics implementations, with demonstrations beyond 3,000 GPUs.

  • Takeaways & Limitations

    GPU-centric design and spatial domain decomposition can support large-scale molecular dynamics while exploiting parallelism across GPUs and nodes.

  • Takeaways & Limitations

    The MPI release does not support multi-GPU electrostatics, rigid bodies, or anisotropic particles and relies exclusively on spatial domain decomposition, limiting applicability to mostly homogeneous systems.

Abstract

from arXiv · show

We describe a highly optimized implementation of MPI domain decomposition in a GPU-enabled, general-purpose molecular dynamics code, HOOMD-blue (Anderson and Glotzer, arXiv:1308.5587). Our approach is inspired by a traditional CPU-based code, LAMMPS (Plimpton, J. Comp. Phys. 117, 1995), but is implemented within a code that was designed for execution on GPUs from the start (Anderson et al., J. Comp. Phys. 227, 2008). The software supports short-ranged pair force and bond force fields and achieves optimal GPU performance using an autotuning algorithm. We are able to demonstrate equivalent or superior scaling on up to 3,375 GPUs in Lennard-Jones and dissipative particle dynamics (DPD) simulations of up to 108 million particles. GPUDirect RDMA capabilities in recent GPU generations provide better performance in full double precision calculations. For a representative polymer physics application, HOOMD-blue 1.0 provides an effective GPU vs. CPU node speed-up of 12.5x.

1. Introduction

HOOMD-blue extends GPU-first molecular dynamics to efficient multi-GPU execution, addressing communication latency and diminishing per-GPU workloads in strong scaling. The paper demonstrates scaling beyond 3,000 GPUs and compares favorably with other GPU-enabled implementations.

  • Motivation: GPU acceleration has made molecular dynamics feasible on desktop workstations, while large GPU clusters require efficient many-GPU utilization.The paper motivates this need using systems such as Titan and Blue Waters.
  • Challenges: Strong scaling is limited by PCIe communication latency, reduced work per GPU, and underutilization as N/P decreases at fixed system size.Communication should be overlapped with computation whenever possible to mask latency.
  • Contribution: HOOMD-blue 1.0 adds MPI spatial domain decomposition and demonstrates scaling on over 3,000 GPUs.The implementation targets the communication and workload challenges of multi-GPU strong scaling.
  • Results: Strong-scaling speed-ups exceeding 50x are demonstrated on Titan, while weak scaling holds across three orders of magnitude in system size.The paper also compares HOOMD-blue with LAMMPS-GPU, LAMMPS USER-MESO, and LAMMPS-Kokkos.
  • Evaluation: The study evaluates Lennard-Jones and polymer-brush scaling, GPUDirect RDMA, and dissipative particle dynamics benchmarks.These evaluations cover both scaling behavior and communication technology choices.

2. Scientific applications

HOOMD-blue’s MPI domain decomposition is intended to support large soft-matter and coarse-grained simulations whose physical structures require large system sizes and many interacting units.

  • Applications: MPI domain decomposition extends HOOMD-blue’s previously single-GPU functionality to polymer systems and complex fluids.The software’s flexibility and customizability support a wide range of soft condensed matter applications.
  • Polymer systems: Bulk polymer liquids require long chains, large simulation cells, and many polymer coils to investigate model-material thermodynamics.Their approximately homogeneous density and effective incompressibility make these systems particularly amenable to domain decomposition.
  • Complex systems: Large-scale simulations are also needed for anisotropic particles, polymer nanocomposites, colloidal crystals, active particles, and biological systems.The structures of interest can be much larger than their constituent particles, requiring many simulated units.

3. Characterization of HOOMD-blue

HOOMD-blue 1.0 preserves its general-purpose, customizable simulation framework while supporting GPU and CPU execution with MPI functionality and broad simulation features.

  • Scope: HOOMD-blue is a general-purpose code primarily targeting soft-matter and coarse-grained simulations.Its existing functionality is preserved in single-GPU and single-core runs.
  • Execution: MPI functionality can use multiple Python interpreters across ranks while retaining GPU and CPU execution paths.The passage states that existing functionality remains available in both execution modes.
  • Architecture: The software is driven by Python scripts, with a CUDA/C++ simulation engine and compile-time full double-precision support.Version 1.0 is optimized for Kepler GPUs and also runs on Fermi GPUs.
  • MPI-enabled features: MPI support required updates to file I/O, integrators, pair and bond potentials, and analyzer classes.Supported ensembles include NVE, NVT, NPH, and NPT.

4. Implementation of the communication algorithm

HOOMD-blue implements GPU-centric MPI domain decomposition using direct device-side communication, GPU packing, dense-array particle management, and all-neighbor nonblocking exchanges.

  • General strategy: The implementation decomposes space across processor grids, communicates ghost particles for short-ranged forces, and migrates particle ownership between neighboring domains.Communication follows the necessary split dimensions and uses a minimum interface-area rule.
  • General strategy: CUDA-aware MPI targets ghost-particle communication, the largest communication bottleneck, and can enable peer-to-peer and GPUDirect RDMA transfers.These capabilities allow MPI to optimize transfers between GPUs or between a GPU and the network interface.
  • GPU-based communication algorithm: The communication algorithm sends and receives particle data directly from device-side structures, with particle packing and unpacking fully implemented on the GPU.This reduces unnecessary device-host data transfers.
  • Particle migration: Particle data uses dense arrays, with stream compaction removing holes after migration and infrequent array reordering during neighbor-list rebuilds.Reordering typically occurs every 5-20 time steps.
  • Scope boundary: Multi-GPU mode supports only short-ranged molecular dynamics force fields.The stated limitation excludes longer-ranged force calculations from the current multi-GPU implementation.
  • Particle migration: Simultaneous nonblocking communication with all 26 neighbors outperforms six directional exchanges by avoiding GPU data dependencies and enabling MPI pipelining.The six-call LAMMPS-style approach incurs extra device-host copies and kernel calls between exchanges.

Ghost particle exchange:

HOOMD-blue’s communication design combines GPU-side particle exchange with domain-aware handling of bonded groups to reduce communication costs while supporting dynamic molecular structures.

  • Particle exchange: Ghost particles are marked, expanded into destination-rank pairs, sorted by rank, and gathered into send buffers.The algorithm permits repeated particle indices when one ghost particle must be sent to multiple neighboring domains.
  • Communication scheduling: Grouping global synchronization separately from neighborhood communication mitigates load imbalance and concentrates synchronization at one point in the time step.This avoids interleaving global synchronization with computation or neighborhood communication.
  • Domain placement: Mapping Cartesian domains to logical MPI ranks reduces expensive inter-node communication during neighborhood exchanges.Domains executed on the same physical node are grouped together.
  • Bonded groups: Bonded groups use generic data structures, allowing bonds, angles, dihedrals, and impropers to share an implementation.Bond data remain dynamic and scale as O(N/P) in memory and computation when migrated with particles.
  • Bonded groups: Bond migration tracks processor ownership of bond members, then replicates bonds and exchanges required ghost particles across split domains.The ownership tables are initialized by neighbor broadcasts and updated after particle migration.
  • Bonded groups: Communication occurs only between neighboring domains, limiting bonds to span at most two domains under periodic boundary conditions.Bond lengths are consequently limited to half the local domain size.

5. Optimizations for strong scaling

Strong scaling is improved by adapting GPU kernel parallelism to shrinking per-GPU workloads and by optimizing communication, especially the recurring ghost update.

  • 5.1. Force computation with a cooperative thread array: Pair-force and neighbor-list computation are parallelized across w cooperative threads per particle, with w constrained by GPU warp execution.The same parameter controls both neighbor-list construction and force computation.
  • 5.1. Force computation with a cooperative thread array: Neighbor indices are compacted within warp segments, while partial forces are combined through intra-warp floating-point reduction.These operations distribute per-particle neighbor work across cooperative threads.
  • 5.2. Autotuner: An autotuner sweeps 160 combinations of block size and w, compares median runtimes from five executions, and periodically fixes the fastest setting.Tuning typically completes in 15,000–20,000 steps before later rescans adjust parameters.
  • 5.3. Scaling of compute kernels vs. communication: Communication time is nearly independent of particles per GPU, so scaling breaks down below approximately 10,000 particles per GPU.The crossover occurs between P = 4 and P = 8 as communication becomes limiting.
  • 5.3. Scaling of compute kernels vs. communication: The ghost update dominates communication time and is optimized by overlapping it with global synchronization and using CUDA-aware MPI.Migration, ghost exchange, and update contributions vary only weakly with particles per GPU in the studied regime.

6. Scaling performance

HOOMD-blue maintains useful weak and strong scaling across thousands of GPUs, with performance roughly twice that of LAMMPS-GPU in the reported Lennard-Jones benchmarks. Strong-scaling efficiency depends on particles per GPU, reaching 70% at about 200,000 particles per GPU.

  • Weak scaling: 32,000 particles per GPU were maintained in weak-scaling tests reaching 108,000,000 particles on 3,375 GPUs.The benchmark used an LJ fluid on Titan and compared HOOMD-blue with LAMMPS-GPU.
  • Weak scaling: About 50% performance decline occurred across more than 3,000 GPUs after the initial drop from one to approximately eight GPUs.The initial decline corresponds to switching from no communication to a three-dimensional 2 × 2 × 2 domain decomposition; further losses were attributed to communication latency from collective MPI calls.
  • Comparison: HOOMD-blue achieved roughly a two-fold speed-up over LAMMPS-GPU in both weak and strong Lennard-Jones scaling tests.The strong-scaling comparison used 10,976,000 particles and up to 2,048 GPUs.
  • Strong scaling: Essentially ideal strong scaling held to 32 GPUs, while useful scaling continued to 1,024 GPUs for the 10,976,000-particle benchmark.Both implementations showed qualitatively similar scaling limits from architecture-inherent communication bottlenecks.
  • Strong-scaling efficiency: 70% strong-scaling efficiency was achieved at approximately 200,000 particles per GPU, while maximum efficiency occurred above roughly 10^6 particles per GPU.The efficiency analysis covered systems from 256,000 to 10,976,000 particles and up to 2,048 GPUs.

7. Polymer brush scaling benchmark

The polymer brush benchmark models a computationally demanding polymer-tethered surface system and evaluates HOOMD-blue across three replicated system sizes, including GPU-versus-CPU scaling.

  • Benchmark system: The benchmark uses a polymer brush with hexane solvent and oleic-acid tethers grafted onto an immobile wall to test performance beyond an LJ liquid.The polymer backbones include bond, angle, and dihedral potentials, and the system is equilibrated in the NPT ensemble.
  • System sizes: The initial configuration is replicated along the wall dimension to produce systems containing 107,520, 430,080, and 1,720,320 polymeric particles.
  • Performance comparison: For the N = 430,080 benchmark, HOOMD-blue achieves an average GPU-versus-CPU node speed-up of approximately 12.5x.The comparison uses HOOMD-blue on a GPU node versus 16 CPU cores on Blue Waters’ XK7 nodes.

8. Strong scaling with GPUDirect RDMA

The study evaluates GPU communication technologies for strong scaling and finds that GPUDirect RDMA improves double-precision performance, while message-size thresholds and communication bandwidth constrain gains.

  • Communication technologies: GPUDirect RDMA transfers data between remote GPUs through the network interface card while completely offloading the CPU from communication.Earlier GPUDirect peer-to-peer transfers were limited to GPUs within a single node.
  • Benchmark setup: The Wilkes benchmark uses N = 64,000 particles on nodes equipped with two NVIDIA Tesla K20c GPUs and a GPUDirect RDMA software stack.
  • Double-precision scaling: In double precision, GPUDirect RDMA outperforms both CUDA-aware MPI and host-memory staging, with CUDA-aware MPI benefiting up to four nodes.
  • Communication limits: A 32KB maximum message-size threshold is reached with at least eight GPUs in single precision or 16 GPUs in double precision for the N = 64,000 LJ benchmark.Above the threshold, the MPI library switches to default pipelined communication.
  • Code comparison: The LJ, LAMMPS-Kokkos, and HOOMD-blue comparisons use a double-precision N = 2,097,152-particle benchmark with matching initial configurations and P = 1...8 GPUs.

9. Strong scaling of a DPD benchmark

The DPD benchmark compares HOOMD-blue with two LAMMPS implementations through strong scaling to 1,024 GPUs, showing similar large-scale performance and communication-bound behavior.

  • Communication pattern: DPD communication sends twice as much data per time step because ghost-particle velocities are required for the drag force.Global particle IDs of ghost particles are also communicated to seed pairwise random-number generators correctly.
  • Benchmark setup: The benchmark contains N = 2,000,000 particles and compares HOOMD-blue with LAMMPS USER-MESO and LAMMPS-GPU through strong scaling to 1,024 GPUs.HOOMD-blue and LAMMPS-GPU use full double precision, whereas USER-MESO uses mixed precision.
  • Scaling results: HOOMD-blue parallels USER-MESO across the GPU range, is approximately 15% superior, and outperforms LAMMPS-GPU at small GPU counts.At larger node counts, performance between the codes is comparable.
  • Scaling limits: The DPD benchmark is less sensitive to latency optimizations because it communicates four times more data than LJ in single precision and is likely bandwidth-bound.

10. Conclusion and Outlook

The paper presents a GPU-centric MPI implementation that scales effectively across many GPUs, while identifying communication bottlenecks and feature limitations that motivate future work.

  • Implementation: HOOMD-blue reduces device-to-device communication costs with GPU communication routines, CUDA-aware MPI, cooperative thread arrays, and runtime kernel autotuning.
  • Scaling results: Across weak and strong scaling benchmarks, HOOMD-blue performs equivalently or better than similarly optimized GPU-enabled molecular dynamics implementations.Its scaling behavior is qualitatively similar to the comparison codes, consistent with limits imposed by the underlying architecture.
  • Outlook: GPUDirect RDMA provides superior double-precision performance in strong-scaling benchmarks but retains current limitations.The authors identify latency and bandwidth bottlenecks as targets for further improvement.
  • Scope boundary: The first MPI release lacks multi-GPU support for electrostatics, rigid bodies, and anisotropic particles, which remain available only in single-GPU simulations.
  • Scope boundary: Spatial domain decomposition primarily applies to homogeneous systems, while more inhomogeneous or biomolecular systems require sophisticated load-balancing schemes.
  • Broader implication: GPU node-level parallelism combined with multi-node scaling establishes parallelism as an enabling strategy for computational physics discovery.
Loading 1412.3387v1…