PaperlayerView exact revision on arXiv

Source-linked AI summary

SAPPORO: A way to turn your graphics cards into a GRAPE-6

Evghenii Gaburov, Stefan Harfst, Simon Portegies Zwart

arXiv:0902.4463v2astro-ph.IMcs.DC

TL;DR

Older NVIDIA GPU architectures lacked double-precision support, complicating high-precision gravitational N-body simulations on that hardware. Sapporo emulates double precision where needed while matching GRAPE-6’s calling sequence; it was twice as fast as commonly used GRAPE6A/GRAPE6-BLX cards and as accurate as GRAPE with standard integration parameters.

  • Problem

    Older NVIDIA GPU architectures lacked double-precision support, constraining high-precision gravitational N-body simulations on that hardware.

    Source: §4 · p. 11

  • Method

    Sapporo emulates double precision in key calculations and matches GRAPE-6’s calling sequence for running existing codes on NVIDIA GPUs.

    Source: §1 · p. 2, §4 · p. 11

  • Results

    Sapporo was twice as fast as commonly used GRAPE6A/GRAPE6-BLX cards and as accurate as GRAPE with standard integration parameters.

    Source: §4 · p. 11

  • Takeaways & Limitations

    GRAPE-6 codes can run on GPUs without source-code modifications by switching to Sapporo.

    Source: §1 · p. 2

    Sapporo’s design for older GPU architectures does not efficiently use all GT200 parallel elements, limiting its speedup on that architecture.

    Source: §4 · p. 12

Abstract

from arXiv · show

We present Sapporo, a library for performing high-precision gravitational N-body simulations on NVIDIA Graphical Processing Units (GPUs). Our library mimics the GRAPE-6 library, and N-body codes currently running on GRAPE-6 can switch to Sapporo by a simple relinking of the library. The precision of our library is comparable to that of GRAPE-6, even though internally the GPU hardware is limited to single precision arithmetics. This limitation is effectively overcome by emulating double precision for calculating the distance between particles. The performance loss of this operation is small (< 20%) compared to the advantage of being able to run at high precision. We tested the library using several GRAPE-6-enabled N-body codes, in particular with Starlab and phiGRAPE. We measured peak performance of 800 Gflop/s for running with 10^6 particles on a PC with four commercial G92 architecture GPUs (two GeForce 9800GX2). As a production test, we simulated a 32k Plummer model with equal mass stars well beyond core collapse. The simulation took 41 days, during which the mean performance was 113 Gflop/s. The GPU did not show any problems from running in a production environment for such an extended period of time.

1. Introduction

Sapporo brings high-precision gravitational N-body simulations to NVIDIA GPUs through a CUDA library that GRAPE-6 codes can use by relinking. It targets GPU accuracy limits while measuring accuracy and performance in Starlab and phiGRAPE.

1. Introduction

  • GPUs were presented as increasingly accessible scientific-computing hardware, with lower cost, footprint, and CO2 production than traditional parallel computers.The passage also notes that local GPU workstations avoid waiting queues for large simulations.Source: §1 · p. 1
  • Earlier GPU N-body implementations were limited by single-precision accuracy, while native double precision on the GTX280 carried a dramatic performance hit.Only a limited number of processor pipelines supported double-precision calculations.Source: §1 · p. 1
  • GRAPE-6 codes can run on NVIDIA GPUs through Sapporo by relinking, without source-code modifications.Sapporo is written in CUDA and closely matches the GRAPE-6 library calling sequence.Source: §1 · p. 2
  • The authors assess Sapporo’s accuracy and performance using two direct N-body simulation environments: Starlab and phiGRAPE.phiGRAPE is implemented in the Multi Scale Software Environment (MUSE).Source: §1 · p. 2
  • Sapporo uses targeted double-precision arithmetic to support high-precision GPU N-body calculations, with a performance loss of ∼20%.The library identifies operations where double precision matters most.Source: §1 · p. 2

2. Implementation

Sapporo splits Hermite N-body work between CPU and GPU, using parallel GPU kernels for prediction and force calculation while retaining high-precision distance calculations. Its implementation supports multi-GPU execution and existing GRAPE-6 applications through a compatible interface.

2. Implementation

  • The implementation targets GPU parallelism while managing memory latency by cooperatively loading global-memory data into shared memory.A G80/G92 GPU has 16 multiprocessors and can execute up to 12,288 threads in parallel.Source: §2 · p. 2

2.1. Task decomposition between the CPU and the GPU

  • The CPU handles communication, i-particle prediction, and correction, while the GPU predicts j-particles and calculates forces, reducing communication to i-particles.Sending all n particles to the GPU at every time step would make communication a bottleneck.Source: §2.1 · p. 3, Figure 1 · p. 3

2.2. Predictor step

  • The GPU predicts j-particle positions and velocities; double-single arithmetic is used where double precision is needed, while other operations remain single precision.Prediction is sensitive to position round-off, so the summation is performed in double precision without requiring hardware support for it.Source: §2.2 · pp. 3–4

2.3. The calculation step

  • Each multiprocessor processes its assigned j-particles against the i-particles, loading shared-memory batches before sequentially accumulating partial forces.If the i-particle set underfills the available threads, dividing the j-particles among additional threads can keep those threads active; partial forces are reduced on the GPU.Source: §2.3 · p. 5
  • Double-single subtraction computes particle separations with three floating-point operations, resolving them to one part in ten million or better.This precision addresses round-off in distances, which can degrade accelerations, jerks, and simulation accuracy.Source: §2.3 · p. 6

2.4. Parallelization over multiple GPUs

  • With Q GPUs, Sapporo divides the j-particles evenly among them while each GPU processes the same i-particles; this speeds up sufficiently large systems but may slow smaller ones.The distribution is automatic and requires no application source-code changes.Source: §2.4 · p. 6

2.5. Interface

  • Sapporo matches the GRAPE-6 API, allowing compatible applications to use it by relinking without source-code changes.It supports per-block softening and nearest-neighbor searches for i-particles, and requires a CUDA-enabled GPU and CUDA runtime libraries.Source: §2.5 · p. 6

3. Results

The tests use Starlab and phiGRAPE on Plummer models, comparing GPU performance and accuracy with GRAPE-6A, and include a 32k cluster evolution through core collapse.

3.1. Performance

  • For N ≤64k, four-GPU speed-up degraded because each GPU had fewer particles per block; similar behavior occurred on the GRAPE-6A cluster.Wall-clock time otherwise scaled as expected with N^2 and decreased as more GPUs were used.Source: §3.1 · p. 7
  • Sapporo sustained about twice the performance of the GRAPE cluster across the tested particle counts.Figure 6 compares computational speed; the authors note that inter-node communication may affect the GRAPE-cluster comparison.Source: Figure 6 · p. 8, §3.1 · p. 8

3.3. Star cluster evolution and core collapse

  • Sustained speed fell from about 140 Gflop/s to 132 Gflop/s at core collapse and 88 Gflop/s afterward, when dense cores and hard binaries required extra host calculations.The mean over the full simulation was about 113 Gflop/s.Source: §3.3 · pp. 9, 11

3.2. Accuracy

  • For η ≳3 · 10^-3, GPU and GRAPE energy errors were indistinguishable; below that, GPU error saturated near 10^-9 versus GRAPE-6A’s 10^-11.Typical η values of 10^-2 to 10^-1 lie above the GPU’s precision-limited regime.Source: §3.2 · p. 9, Figure 8 · p. 10

3.3. Star cluster evolution and core collapse

  • The 32k equal-mass Plummer simulation reached core collapse at t ≃6440 and continued to t ≃9200 over about 41 days.Mean sustained performance over the full calculation was about 113 Gflop/s.Source: §3.3 · pp. 9, 11, Figure 9 · p. 10

4. Summary

Sapporo emulates GRAPE-6 functionality on NVIDIA GPUs and lets existing GRAPE-6-interface programs run without source changes. Tests found performance twice that of commonly used GRAPE-6 cards, comparable accuracy, and a roughly 20% performance cost for double-precision emulation.

4. Summary

  • Sapporo emulates GRAPE-6 functionality and interface, allowing existing GRAPE-6-interface programs to run on GPUs by relinking, with some rarely used functions excepted.The library can be configured to use combinations of available GPUs without application source changes.Source: §4 · p. 11
  • Sapporo was twice as fast as commonly used GRAPE6A/GRAPE6-BLX cards and as accurate as GRAPE with standard integration parameters.A PC with two GeForce 9800GX2 cards was also on par with an eight-node GRAPE6A cluster or a 32-chip GRAPE6 board.Source: §4 · p. 11
  • Sapporo splits Hermite integration between GPU prediction and force calculation and CPU correction, keeping force calculation dominant.The authors report that host-side prediction instead would make prediction and communication dominate and degrade performance.Source: §4 · p. 11
  • Neighbor lists support nearest-neighbor functionality, but calculating and storing them costs a few tens of percent in performance depending on the configured maximum.Disabling neighbor lists can gain up to 50% in speed.Source: §4 · p. 11
  • Emulating double precision with single-precision numbers costs about a 20% performance loss while supporting high-precision force calculations.The library places double precision in operations where it is most important.Source: §4 · p. 11
  • GT200 supports native double precision only on a limited number of pipelines, where performance is an order of magnitude smaller.The library's double-precision emulation was needed because the earlier G80/G92 generation did not support it.Source: §4 · p. 11
  • On GT200 GPUs, Sapporo achieved about a 50% speedup, below the factor-of-two peak-performance increase, because the current version did not efficiently use all parallel elements.The authors were working on a version intended to scale across future NVIDIA GPU generations, including GT200.Source: §4 · p. 12
Loading 0902.4463v2…