Source-linked AI summary
Grid: A next generation data parallel C++ QCD library
Peter Boyle, Azusa Yamaguchi, Guido Cossu, Antonin Portelli
TL;DR
Grid is a new QCD physics library motivated by increasingly parallel supercomputer architectures and the difficulty of efficiently programming them. It combines high-level data parallelism with C++11-based vector abstractions and reports encouraging performance, including 65% of peak on cache-resident data and about 40% of quad-core peak for selected operators. The code remains immature and is still a work in progress.
Problem
Increasingly parallel architectures make efficient programming difficult, motivating a refreshed QCD code base that supports multiple forms of parallelism.
Method
Grid combines high-level data parallel constructs with C++11 vector abstractions and compile-time specialization to support portable MPI, OpenMP, and SIMD performance.
Results
Grid achieves 65% of peak on cache-resident data and around 40% of quad-core peak for Wilson and Domain Wall fermion operators.
Takeaways & Limitations
Grid provides a performance-portable and maintainable foundation with substantial functionality already implemented for future large-scale chiral-fermion simulations.
Takeaways & Limitations
Grid is new, growing, not yet mature, and remains a work in progress.
Abstract
from arXiv · showhide
In this proceedings we discuss the motivation, implementation details, and performance of a new physics code base called Grid. It is intended to be more performant, more general, but similar in spirit to QDP++\cite{QDP}. Our approach is to engineer the basic type system to be consistently fast, rather than bolt on a few optimised routines, and we are attempt to write all our optimised routines directly in the Grid framework. It is hoped this will deliver best known practice performance across the next generation of supercomputers, which will provide programming challenges to traditional scalar codes. We illustrate the programming patterns used to implement our goals, and advances in productivity that have been enabled by using new features in C++11.
1. The evolution of computer architectures
Modern supercomputers combine increasingly wide and numerous execution resources with MPI, threading, and SIMD, making efficient programming difficult. Grid addresses this challenge through high-level data parallelism and consistent support for multiple parallelism forms.
- Hardware evolution: Modern systems combine message passing, threading, and SIMD vector parallelism, but lack a universal programming approach.The US roadmap systems cited employ many-core processors with wide vector lengths.
- Hardware evolution: O(4000) operations per cycle increase the programmer’s burden because execution units grow faster than individual-unit speed.Table 1 attributes this challenge to growing parallel execution resources and flops per clock.
- Grid’s response: Grid uses high-level data parallel constructs to target MPI, OpenMP, and short-vector parallelism across multiple instruction-set architectures.Supported instruction sets include SSE, AVX, AVX2, FMA4, IMCI, and AVX512; accelerator offload and ARM Neon were planned.
- Grid’s response: Grid hides subdivisions across underlying parallelism forms, relieving programmers from scheduling details while efficiently supporting threading and SIMD.This is presented as a key difference from QDP++ alongside the existing QCD use of MPI domain decomposition.
- Programming constraints: Existing QCD packages require reengineering for SIMD, while hybrid MPI and OpenMP can reduce redundant copies and MPI memory overhead.The need is driven by many-core systems with up to 32 cores per node and by architecture-dependent SIMD locality constraints.
2. Grid design patterns
Grid abstracts architecture-specific SIMD operations through C++11 vector types and templates, while adapting vector lengths and data layouts to each target. Its design combines portable high-level code with efficient matrix-vector execution, automated tensor handling, serialization, and expression-template fusion.
- SIMD abstraction: Grid localizes architecture-dependent intrinsic sequences behind overloaded inline vector operators, keeping most code platform independent.The vector classes support scalar and target-specific implementations while exposing a common interface.
- SIMD abstraction: Vector length is obtained from Nsimd() and used to parameterize optimizations and layout transformations, rather than fixed independently of the target architecture.
- SIMD abstraction: Porting Grid to a new SIMD architecture requires about 300 lines of code.
- Matrix-vector code: Interleaving multiple small matrix-vector products across SIMD lanes lets one templated routine perform four, eight, or sixteen products without horizontal reductions.The same routine accepts scalar or Grid vector types, with the vector width determining the number of simultaneous products.
- Matrix-vector code: Compile-time matrix dimensions enable automatic loop unrolling, with cache-resident implementations reaching about 65% of peak performance on modern Intel processors.
- C++11 productivity: C++11 templates generalize scalar, vector, and matrix operations, while expression templates fuse lattice operations and reduce implementation code volume.Grid also generates serialisation code for parameter classes and supports multiple file representations through its virtualized scheme.
3. Performance
Grid achieves strong performance across cache-resident and bandwidth-limited SU(3) matrix multiplication, while also delivering substantial performance on fermion operators and portability across compilers and architectures.
- SU(3) matrix multiplication: 65% of peak performance is achieved on cache-resident SU(3) matrix multiplication, while out-of-cache performance slightly exceeds STREAM bandwidth.The benchmark uses one Intel Core i7-3615QM core and compares Grid with QDP++ compiled using g++ v4.9 without SSE kernels.
- SU(3) matrix multiplication: QDP++ is around 5.5 times slower than Grid for the same SU(3) matrix multiplication on the same chip.QDP++ did not compile with Clang and did not saturate memory bandwidth sufficiently to benefit from cache.
- Fermion operators: Around 40% of the 147 GF peak is achieved by Grid on Wilson and Domain Wall fermion operators using a quad-core Ivybridge.Imbalanced multiples and adds reduce the maximum achievable performance to 78%.
- Memory systems: The Crystalwell comparison uses AVX-1 to hold peak performance roughly fixed while exposing the benefit of its 128MB on-package eDRAM cache over Ivybridge.Grid also provides an AVX-2 target that uses multiply-add instructions where possible.
- Compiler and architecture portability: Grid performance remains strong across later GCC versions and ICC, varying somewhat by compiler, especially when Nc loops are hand unrolled.The study also compares different generations of Intel and AMD architectures.
4. Status, conclusions and outlook
Grid has implemented substantial QCD functionality and reports encouraging prospects for performance-portable simulations, while remaining an immature work in progress. Its development targets large-scale 5D chiral fermion simulations and is facilitated by a high-level programming environment.
- Status and limitations: Grid remains a new, growing, and not-yet-mature physics system under active development.The code is described as a work in progress and is developed through a public repository.
- Performance evaluation: Figure 6 compares domain-wall dslash code-generation quality across compilers, including hand-unrolled and Nc-loop variants.The passage notes that Nc-loop variants are slower for Intel icpc V16.0, while Clang++ performs reasonably without unrolling.
- Outlook: Grid targets outstanding performance for next-generation 5D chiral fermion simulations on the largest systems.The stated target period is 2016–2020.
- Implemented functionality: Significant functionality has been implemented, including conjugate-residual solvers, five-dimensional chiral fermions, periodic boundary conditions, and nested integrators.The listed functionality also includes G-parity boundary conditions.
- Development: Development is rapid and is facilitated by the high-level programming environment.