PaperlayerView exact revision on arXiv

Source-linked AI summary

Best bang for your buck: GPU nodes for GROMACS biomolecular simulations

Carsten Kutzner, Szilárd Páll, Martin Fechner, Ansgar Esztermann, Bert L. de Groot, Helmut Grubmüller

arXiv:1507.00898v1cs.DCcs.PFphysics.bio-phphysics.comp-phq-bio.BM

TL;DR

Choosing economical GROMACS hardware requires balancing trajectory throughput against hardware, electricity, and cooling costs. This study benchmarks CPU/GPU node configurations across performance and cost criteria, finding that consumer-GPU nodes offer the strongest performance-to-price results, subject to their hardware limitations.

  • Problem

    The paper asks which hardware configuration produces the most GROMACS molecular-dynamics trajectory for a fixed investment in hardware, electricity, and cooling.

  • Method

    The authors benchmark representative biomolecular simulations across CPU/GPU node combinations and evaluate trajectory performance, performance-to-price, parallel performance, energy, and rack-space criteria.

  • Results

    Nodes with consumer-class GPUs achieve the best performance-to-price ratios, with at least one consumer GPU increasing node performance relative to nodes without GPUs.

  • Takeaways & Limitations

    For GROMACS, nodes with consumer-class GPUs are the most economic choice within the study’s stated limitations.

  • Takeaways & Limitations

    The hardware recommendations are specific to biomolecular MD and GROMACS and may not suit general-purpose clusters serving broad application workloads.

Abstract

from arXiv · show

The molecular dynamics simulation package GROMACS runs efficiently on a wide variety of hardware from commodity workstations to high performance computing clusters. Hardware features are well exploited with a combination of SIMD, multi-threading, and MPI-based SPMD/MPMD parallelism, while GPUs can be used as accelerators to compute interactions offloaded from the CPU. Here we evaluate which hardware produces trajectories with GROMACS 4.6 or 5.0 in the most economical way. We have assembled and benchmarked compute nodes with various CPU/GPU combinations to identify optimal compositions in terms of raw trajectory production rate, performance-to-price ratio, energy efficiency, and several other criteria. Though hardware prices are naturally subject to trends and fluctuations, general tendencies are clearly visible. Adding any type of GPU significantly boosts a node's simulation performance. For inexpensive consumer-class GPUs this improvement equally reflects in the performance-to-price ratio. Although memory issues in consumer-class GPUs could pass unnoticed since these cards do not support ECC memory, unreliable GPUs can be sorted out with memory checking tools. Apart from the obvious determinants for cost-efficiency like hardware expenses and raw performance, the energy consumption of a node is a major cost factor. Over the typical hardware lifetime until replacement of a few years, the costs for electrical power and cooling can become larger than the costs of the hardware itself. Taking that into account, nodes with a well-balanced ratio of CPU and consumer-class GPU resources produce the maximum amount of GROMACS trajectory over their lifetime.

1 Introduction

The study asks which hardware configurations run GROMACS most economically and evaluates performance across representative biomolecular systems and CPU/GPU combinations. It frames optimality through trajectory production per investment and other performance, cost, energy, and space criteria.

  • GROMACS performance and hardware: GROMACS combines SIMD, OpenMP/thread-MPI, MPI, and GPU offloading to exploit heterogeneous hardware.GPUs can compute short-range non-bonded forces while CPUs handle remaining forces and particle updates.
  • Research question: The central question is which hardware is optimal for GROMACS and how to obtain optimal performance.The study focuses on configurations that produce trajectories more economically than others.
  • Study design: The evaluation measures representative biomolecular systems on hardware combinations with and without GPU acceleration, tuning run parameters for comparable numerical accuracy.The study also measures power consumption, trajectory production costs, and highly parallel cluster performance.
  • Evaluation criteria: Optimality can involve performance-to-price, single-node performance, time-to-solution, energy-to-solution, and rack-space requirements.Researchers may weight these criteria differently depending on their projects.
  • Practical objective: For a fixed budget covering hardware, electricity, and cooling, the target is the largest amount of MD trajectory per investment.The study provides benchmark expectations, input files, and settings for tested node types.
  • Scope: GROMACS-focused hardware may not be best for a general-purpose cluster serving a broad range of applications.The authors note that a middle-ground can provide good performance for GROMACS and other applications.

2 Key determinants for GROMACS performance

GROMACS automates much of resource placement on single nodes, but multi-CPU/GPU and cluster runs benefit from tuning rank, thread, domain, and load-balancing choices. Performance depends on balancing CPU and GPU work while avoiding decomposition and thermal bottlenecks.

  • Automatic resource placement: mdrun automatically detects hardware resources and can select thread counts, ranks, layouts, and GPU-to-rank assignments.These heuristics often achieve near-optimal performance for single-node runs with one CPU and one GPU.
  • Parallel decomposition: PME task decomposition reduces costly 3D-FFT all-to-all communication by separating PME ranks from particle-processing ranks.For many ranks, peak performance is usually reached with an appropriate Nrank = NDD + NPME separation.
  • CPU-GPU work division: GPU acceleration offloads short-range Coulomb and van der Waals interactions, while CPUs compute long-range PME and bonded interactions.Increasing the short-range cutoff and PME grid spacing shifts computational load between PP and PME resources while preserving comparable accuracy.
  • Domain decomposition: Using fewer, larger domains can reduce GPU-offload overhead, whereas multiple narrower ranks per GPU can reduce threading, cache-coherency, and inter-socket overheads.The better choice depends on the relative numbers of CPUs, NUMA regions, GPUs, and system size.
  • Load balancing: Dynamic load balancing can conflict with CPU-GPU balancing when domain cells approach the cutoff size, limiting cutoff scaling and harming performance.The authors describe ongoing work to eliminate this load-balancer interplay.
  • Thermal behavior: Thermal throttling reduced a GeForce GTX TITAN clock speed by 7% and can cause load-balancing problems and application slowdown.Performance gains from thermal headroom are largest in strongly GPU-bound simulations and smaller in balanced or CPU-bound cases.

3 Methods

The benchmarks use two representative biomolecular systems and systematically test GROMACS across CPU/GPU nodes, software environments, precision modes, GPU architectures, and launch settings. Measurements include short runs with warm-up steps excluded and repeated GPU tests.

  • Benchmark systems: The benchmark systems are MEM, an approximately 80 k-atom membrane channel system, and RIB, a bacterial ribosome exceeding two million atoms.MEM represents membrane-protein setups, while RIB represents a large system typically run across several nodes.
  • Hardware and software: The study tests GROMACS 4.6 across varied node types and GPU combinations, with version-5.0 relevance expected where kernel and parallelization designs remain similar.Single-node tests use thread-MPI; multi-node tests use Intel or IBM MPI libraries.
  • Precision: Mixed precision was used because it preserves energy conservation and produces 1.4–2 × more trajectory in the same compute time than double precision.GPU runs use mixed precision because GROMACS GPU arithmetic is single precision for the relevant calculations.
  • GPU characterization: GPU comparisons consider theoretical single-precision throughput, sustained frequency, architecture, pricing, memory, and possible frequency throttling.Maxwell GM204 cards can outperform Kepler cards at comparable flop rates because of instruction scheduling and latency differences.
  • Measurement procedure: The benchmarks run 2,000–15,000 steps, exclude the first 1,000–10,000 steps during load balancing, and average two runs for MEM and RIB GPU results.Reported standard deviations are on the order of a few percent.
  • Parameter tuning: Launch settings are optimized by testing rank/thread combinations, PME-rank tuning, Hyper-Threading, dynamic load balancing, and GPU-sharing options.The tested combinations include Nrank × Nth = Nc for nodes with Nc cores.

4 Results

The results assess GPU reliability, compiler effects, performance settings, cost efficiency, and energy-related throughput across GROMACS node configurations. Balanced CPU-GPU resources and concurrent simulations improve GPU-node efficiency and throughput.

  • Evaluation scope: The study derives trajectory produced per invested euro, compares nodes with and without GPUs, evaluates clock dependence and energy efficiency, and examines strong scaling.It also tests parallelization settings and concurrent simulations, finding that balanced CPU-GPU resources support high efficiency and that concurrent runs maximize GPU-node throughput.
  • GPU memory reliability: Consumer-class GPU reliability can be screened with memory-checking tools despite the absence of ECC memory.The paper notes that rare memory bit-flips could cause silent corruption and incorrect simulation results in a worst-case scenario.
  • GPU memory reliability: Eight consumer-class GPUs showed memory errors during stress testing, while the vast majority of tested cards were error-free.The tests allocated all available GPU memory and ran for at least 4,500 iterations; detected problematic cards were replaced.
  • GPU memory reliability: >50 million errors were registered for one 780Ti over 10,000 iterations, whereas some cards showed only a couple of errors over the same duration.The first iteration already detected more than 1,000 errors in the most problematic case, and error rates were nearly constant across repeats.
  • Compiler choice: GCC 4.8 produced the fastest executable on both AMD and Intel platforms.Compiler choice affected GPU-node performance by at most 4%, but the difference reached 20% without GPUs.

Impact of neighbor searching frequency

Neighbor-search frequency controls the trade-off between list-construction cost and short-range-force computation, while also affecting CPU-GPU overlap. In the tested GPU configuration, the best frequency was between 20 and 70 time steps, with 40 used for benchmarks.

  • Performance mechanism: Reducing search frequency trades neighbor-list construction cost against short-range-force computation and increases average CPU-GPU overlap.The GPU is idle during CPU-based list construction, making less frequent searches especially relevant for accelerator utilization.
  • Performance mechanism: In multi-GPU runs, decreasing search frequency can increase overlap when domain decomposition occurs on the same step as neighbor searching.The effect follows from coordinating CPU list construction with GPU work and domain decomposition.
  • Frequency choice: The performance optimum for neighbor searching was between 20 and 70 time steps.Performance deteriorated quickly at frequencies of 20 time steps or fewer.
  • Benchmark setting: The benchmarks used a neighbor-search frequency of 40 time steps on GPU nodes.This value lies within the reported optimum range.
  • Figure 2: Figure 2 varies neighbor-search frequency on a node with 2×E5-2680v2 processors and 2×K20X GPUs, also varying ranks and threads per rank for MEM.The figure therefore combines frequency effects with selected parallelization settings in the MEM benchmark.

Influence of hybrid parallelization settings and DLB

GROMACS performance depends on how MPI ranks and OpenMP threads are mixed, with the best combination varying by hardware and MD system. GPU runs generally favor hybrid parallelism, while DLB can either degrade or preserve performance depending on the case.

  • CPU-only parallelization: CPU-only benchmarks achieved highest performance with pure MPI parallelization.This pattern is often observed on nodes without GPUs.
  • Hybrid parallelization: GPU-equipped multi-socket or multi-GPU nodes usually reached highest performance with hybrid parallelism at about 4–5 threads per MPI rank.MPI and OpenMP paths scale differently, so the optimum mix depends on the hardware and simulated system.
  • Hybrid parallelization: 52 ns/d was achieved with 4 MPI ranks and 10 threads each, compared with 24 ns/d using 40 MPI ranks in the single-GPU MEM setting.The 40-rank configuration delivered less than half the performance of the optimal settings.
  • Figure 3: Figure 3 compares single-node performance across GPU counts and MPI/OpenMP allocations, with solid lines for DLB enabled and dotted lines for DLB disabled.The test node contained 2× E5-2680v2 processors and 4× GTX 980+ GPUs, with MEM and RIB shown in separate panels.
  • DLB: DLB caused performance degradation in some GPU cases, including the highest RIB performances in Figure 3 when GPUs were used without DLB.Other cases, such as the 4-GPU MEM setting, showed similar performance with and without DLB.

Fitness of various node types

Node fitness is evaluated using performance, approximate cost, performance-to-price ratio, and launch configurations across MEM and RIB benchmarks. The best settings depend on CPU and GPU composition, with multiple ranks per GPU often advantageous for multi-GPU nodes.

  • Cost assumptions: Retail prices can vary by 15–20% over a relatively short period, limiting the precision of the node cost estimates.The reported cost column reflects rough net prices from 2014 and excludes the InfiniBand network adapter.
  • Cost efficiency: The performance-to-price ratio normalizes hardware results against the lowest-performing setup to identify more competitive node combinations.The authors describe this ratio as approximate but still informative for comparing hardware combinations.
  • Node configurations: Single CPUs with 4–6 physical cores paired with one GPU performed best using threading without domain decomposition.On 10-core CPUs, peak performance was usually obtained with thread-MPI and multiple threads per rank.
  • Node configurations: Multiple-GPU configurations generally favored an even larger number of ranks, with at least Nrank = NGPU ranks required.The benchmark tables report the corresponding DD-grid, thread, rank, and DLB parameters for peak performance.
  • Benchmark coverage: The node-performance tables report single-node MEM and RIB performance across diverse hardware combinations.Table 7 uses the same format as Table 6 but reports the RIB benchmark.

Speedup with GPUs

GPUs substantially increase GROMACS node performance, with consumer-class GPUs also improving performance-to-price ratios. Balanced CPU–GPU configurations outperform more extreme configurations economically, while higher GPU clocks provide modest additional gains.

  • 1.7–3.8×: GPUs increase compute-node performance across tested configurations.
  • 2–3×: adding at least one GPU increases the performance-to-price ratio.
  • Additional GPUs provide less than 20% benefit on the 80 k atom system for single-CPU nodes.
  • ≈67 ns/d for MEM and ≈5 ns/d for RIB: four GTX 980+ GPUs deliver the highest tested single-node performance.
  • Two GPUs are the performance-to-price sweet spot for 20-core E5-2680v2 nodes.

Energy efficiency

Energy consumption materially changes the hardware ranking: one or two GPUs minimize trajectory costs, while Maxwell GPUs are more energy-efficient than Kepler cards. CPU and GPU resources must be balanced because extra GPUs can become CPU-limited.

  • 1.5–2×: nodes with one or two GPUs produce more MD trajectory per invested euro than CPU-only nodes.
  • Third or fourth GPUs significantly raise trajectory cost because runs become CPU-bound, although costs remain below CPU-only levels.
  • ≈20%: Maxwell cards reduce trajectory costs compared with Kepler, with the lowest cost for two E5-2670v2 CPUs and one GTX 980.
  • 52% (44%): replacing E5-2670v2 with E5-2680v2 CPUs increases energy consumption for one GPU (two GPUs), while trajectory costs rise 30% (21%).
  • 2.1×: offloading short-range pair interactions and shifting work from PME calculations to GPU kernels increases performance over no GPU.
  • More or faster GPUs have little effect when the bottleneck remains on the CPU side.

Multi-simulation throughput

Running multiple independent simulations per node improves aggregate throughput by using CPU and GPU resources more efficiently. The benefit is greatest for smaller systems and configurations with weaker single-simulation overlap.

  • Multiple replicas improve parallel efficiency and GPU utilization through GPU sharing and task overlap.
  • 47%: five replicas on a node with one GTX 980 exceed the single-simulation optimum.
  • 40%: five replicas on a node with two GTX 980 GPUs exceed the optimal single-simulation performance.
  • 87%: five replicas with two GTX 980 GPUs exceed a single simulation using default settings.
  • Benefits increase for smaller systems, more CPU cores per GPU, and worse single-simulation CPU–GPU overlap.
  • Nearly 2.5×: multi-simulation throughput increases for the 8 k atom VIL system on the 16-core AMD node.

Strong scaling

Strong scaling can deliver very high raw throughput but sacrifices parallel efficiency, making achieved efficiency the key constraint on performance-to-price ratio. Hardware topology and MPI configuration materially affect multi-node results.

  • Using thread-MPI on single nodes and standard MPI across nodes produces a more pronounced observed efficiency drop than standard MPI throughout.
  • Parallel efficiency, rather than raw performance alone, is the main factor limiting performance-to-price ratio across multiple nodes.
  • Over 300 ns/d for MEM and over 200 ns/d for RIB are achieved on state-of-the-art hardware, at the expense of parallel efficiency.
  • Separate PME ranks and homogeneous interleaved PME distributions improve GPU-node performance at large node counts.
  • 5–15%: hyper-threading performance benefits are largest on individual nodes and in low- to medium-scale regimes.
  • E3-1270v2 GPU nodes have attractive performance-to-price ratios but are poorly suited to parallel operation because of limited PCIe lanes.

5 Discussion

Consumer-class GPUs provide the strongest economic performance for GROMACS, while balanced CPU/GPU configurations maximize lifetime trajectory production when energy costs are included. Parallel efficiency, hardware settings, reliability, and deployment constraints determine the practical optimum.

  • Hardware comparison: Consumer GPUs deliver the best performance-to-price ratios, whereas dual-CPU nodes with multiple GPUs achieve the highest single-node performance.Consumer-GPU nodes are shifted toward higher performance-to-price than GPU-less or Tesla-GPU nodes.
  • Hardware comparison: Adding a consumer-grade GPU increases performance-to-price by about twofold, while HPC GPUs are more expensive and less productive with GROMACS.The comparison given is Tesla K20X instead of GeForce GTX 980.
  • Hardware comparison: Consumer PCs with GeForce GPUs costing around 1,000 e produce the largest amount of MD trajectory per money spent.The optimal performance depends on the ratio between CPU and GPU compute power, and these systems lack ECC memory.
  • Energy efficiency: Energy and cooling can dominate lifetime budgets: GPU nodes may draw twice the power but increase GROMACS performance by an even larger factor.With energy costs included, balanced CPU/GPU resources produce the largest lifetime trajectory output.
  • Parallel scaling: For parallel simulations, performance-to-price mainly follows parallel efficiency; consumer-GPU nodes on QDR IB have the highest ratios up to about eight nodes.The lowest-latency interconnect gives the highest parallel performance, but trajectories can cost 2–8× more than on the best single nodes.
  • GROMACS tuning: Configuration choices can change performance substantially: unfavorable parallelization can halve single-node performance, and GCC ≥4.7 yields binaries up to 20% faster.All hardware threads work best on tested single nodes with HT, whereas one thread per physical core performs better at higher node counts.

Supporting Information

The supporting information provides materials for optimizing GROMACS performance and the simulation inputs used in the study.

  • The supporting information includes examples and scripts for optimizing GROMACS performance.
  • Together, these materials support performance optimization and reproduction of the simulation setups.
  • It also contains input .tpr files for the simulation systems.
Loading 1507.00898v1…