Source-linked AI summary
High-Performance Derivative Computations using CoDiPack
Max Sagebaum, Tim Albring, Nicolas R. Gauger
TL;DR
Existing reverse-mode AD tools use different taping strategies, while current implementations may lack needed features, flexibility, or suitability for large-scale software. CoDiPack addresses this gap with an open-source, modular tool centered on Jacobi taping, expression templates, and recursive data structures. The paper reports efficient performance on a generic PDE example and SU2, including black-box parallel differentiation of SU2.
Problem
Existing AD implementations either are closed source or lack essential features and flexibility for large-scale software.
Method
CoDiPack combines Jacobi taping with expression templates and a modular, recursive layout that supports additional taping approaches and efficient operator-overloading AD.
Results
The paper reports high performance for CoDiPack on a generic PDE example and for black-box, parallel differentiation of the complex SU2 CFD code.
Takeaways & Limitations
CoDiPack provides an open-source basis for efficient AD of large-scale and industrial-grade software while retaining an easy-to-use interface.
Takeaways & Limitations
Index reuse reduces adjoint-vector size but prevents ActiveReals using that technique from supporting C-like memory operations.
Abstract
from arXiv · showhide
There are several AD tools available, which all implement different strategies for the reverse mode of AD. The major strategies are primal value taping (implemented e.g. by ADOL-c) and Jacobi taping (implemented e.g. by adept and dco/c++). Especially for Jacobi taping, recent advances by using expression templates make this approach very attractive for large scale software. The current implementations are either closed source or miss essential features and flexibility. Therefore, we present the new AD tool CoDiPack (Code Differentiation Package) in this paper. It is specifically designed for a minimal memory consumption and optimal runtime, such that it can be used for the differentiation of large scale software. An essential part of the design of CoDiPack is the modular layout and the recursive data structures, which do not only allow the efficient implementation of the Jacobi taping approach, but will also enable other approaches like the primal value taping or new research ideas. We will also present the performance value of CoDiPack on a generic PDE example and on the SU2 code.
1. INTRODUCTION
CoDiPack addresses the need for efficient, flexible reverse-mode AD in large-scale software by combining Jacobi taping with expression templates and a modular, open-source design. The paper presents its implementation and evaluates it on a generic PDE example and SU2.
- Motivation: Reverse-mode operator-overloading AD must store primal values or Jacobian information because evaluation reverses the program’s data flow.These alternatives are called primal taping and Jacobi taping, respectively.
- Related tools: ADOL-C implements primal taping but has runtime and memory overhead on large-scale problems, despite its maturity and robustness.The paper presents it as useful for validation and verification during development of new operator-overloading AD tools.
- Related tools: Jacobi taping combined with expression templates can improve runtime efficiency, but existing tools lack important features or use proprietary licensing.The paper identifies adept as open-source but feature-limited and dco as feature-complete but proprietary.
- CoDiPack goals: CoDiPack targets high-performance computing and industrial-grade software while maintaining an open-source philosophy.Its stated development priorities include efficiency, usability, and extensibility.
- CoDiPack design: CoDiPack uses a modular implementation so new taping strategies, high-level operations, and memory layouts can be modified or added.The paper examines how this structure affects performance and applies the tool to a generic PDE example and SU2.
2. ALGORITHMIC DIFFERENTIATION
The paper models program evaluation as a sequence of local statements and derives forward and reverse interpretations through the chain rule. These interpretations compute Jacobian-vector and transposed-Jacobian-vector products, supporting the taping strategies used by CoDiPack.
- Program representation: A numerical evaluation is represented as a sequence of l local statements, each depending directly on preceding variables.The resulting mathematical representation unrolls loops and may contain many statements for large applications.
- Program representation: The state-space formulation composes statement transformations that update one intermediate variable while preserving the others.The state includes inputs, intermediate values, and outputs, with projection matrices selecting input and output components.
- Forward mode: Forward-mode AD evaluates tangent matrix-vector products alongside the primal computation to obtain the Jacobian multiplied by an arbitrary direction.The tangent interpretation follows the same statement order as the original evaluation.
- Reverse mode: Reverse-mode AD evaluates transposed-Jacobian matrix-vector products by traversing the statements in reverse order.The adjoint interpretation accepts an arbitrary seed vector and applies the corresponding transposed Jacobian product.
- Jacobi taping: Jacobi taping stores the gradient of each statement with respect to its directly preceding variables for reverse evaluation.This approach stores local Jacobian information rather than only primal values.
3. EXPRESSION TEMPLATES
Expression templates represent complete statements as recursive expression structures, allowing CoDiPack to compute reverse-mode Jacobians for whole statements instead of splitting them into many binary operations. This reduces stored Jacobians and statements while supporting recursive gradient propagation through the expression.
- C++ operator overloading supports only unary and binary operators, so naïvely handling larger statements splits them into multiple operations.
- 7 Jacobians and 4 statements are required for the split form, whereas the original statement requires 4 Jacobians and 1 statement.Treating the whole statement at once is therefore more efficient in this example.
- Expression templates change operators to compose expression objects, representing an entire statement as a recursive structure.The expression structure can encode nested operations such as POW<MULT<ADD<Real, Real>, SUB<Real, Real>>>.
- The complete expression structure lets AD operate on the whole statement rather than separately processing each binary operation.
- Reverse-mode differentiation computes argument Jacobians and recursively propagates adjoint values through the expression tree.The calcGradient mechanism uses derivatives with respect to each argument and recursive calls for the remaining propagation.
4. DESIGN AND LAYOUT OF CODIPACK
CoDiPack combines expression templates, modular tape interfaces, and recursive data structures to implement efficient and extensible Jacobian taping. Its design targets low memory use and runtime overhead while supporting multiple differentiation modes and configurable memory layouts.
- Design goals: CoDiPack carefully controls stored data because its primary goals are high speed and memory efficiency for HPC applications.The design considers both the data stored and its memory layout.
- Jacobian taping: Each elemental operation is identified by a global counter, allowing adjoint variables to be accessed after overloaded values leave scope.The counter increments whenever a statement is stored, and its current value identifies the left-hand-side value.
- Jacobian taping: Linear indexing reduces per-statement storage from 12 ∗ k + 5 to 12 ∗ k + 1 bytes by omitting the left-hand-side index.Reverse evaluation follows the stored statements in reverse order, so the left-hand-side index can be reconstructed by decrementing.
- Expression templates: Expression-template interfaces compute operation derivatives recursively and support non-double calculation types, including higher-order derivatives.Unary and binary operator templates provide reusable implementations while preserving optimization options for different operator cases.
- Tape interfaces: Separated ActiveReal and tape interfaces let tape implementations store statements through store and collect Jacobian information through recursive calcGradient calls.The user-facing reverse-mode interface includes registerInput, registerOutput, evaluate, setActive, and setPassive.
- Tape storage: A generalized chunk vector manages multiple tape data streams with different running indices as one recursively structured object.Adding a new stream requires inserting a vector into the recursive structure without adding separate management logic.
- Index reuse: Index reuse shrinks the adjoint vector from the total variable count to the maximum number of simultaneously used variables, usually accelerating reverse evaluation.The technique requires every ActiveReal to have a distinct index and prevents c-like memory operations.
5. TESTS
The tests evaluate CoDiPack on a coupled Burgers PDE and a large SU2 aerodynamic case under controlled single- and multi-process configurations. The SU2 application uses AD to construct a discrete adjoint scheme for optimization-related flow analysis.
- Coupled Burgers equation: The coupled Burgers equation provides a lightweight test for rapidly evaluating how CoDiPack changes affect performance.Both default and specially configured CoDiPack settings are tested.
- Coupled Burgers equation: The Burgers test uses the unit square domain, initial solution values as inputs, and the norm of the final solution as the output.The equations are discretized with an upwind finite-difference scheme and use boundary conditions from the exact solution.
- Test configurations: Single-process tests reduce memory-bandwidth limitations and better expose computational performance, whereas multi-process tests run one process on each of 16 cores.All methods are inlined and required memory is allocated before timing on a 16-core node with 128 GB of memory.
- SU2: The SU2 application targets numerical optimization with PDE constraints, where AD supports efficient discrete adjoint methods for complex computational-fluid-dynamics algorithms.CoDiPack was used in SU2 to generate a flexible and robust discrete adjoint solver for Reynolds-averaged Navier–Stokes equations.
- SU2: Applying AD to the code computing J and G constructs the adjoint fixed-point scheme, with Jacobians taped once at the final fixed-point iteration.The interpretation time determines the overall runtime because the right-hand side is repeatedly evaluated at the fixed point.
- SU2: The SU2 testcase is inviscid flow over the LM1021 supersonic aircraft with 5,730,841 interior elements and 214,586 boundary elements.It uses a central spatial scheme with artificial dissipation and explicit Euler pseudo-time stepping.
6. RESULTS
CoDiPack’s performance depends strongly on memory bandwidth, tape complexity, block size, and the presence of transcendental functions. In SU2, preaccumulation substantially reduces interpretation time, overall adjoint runtime, and memory consumption.
- Coupled Burgers equation: The single Burgers configuration runs approximately 80% faster than the multi-process configuration because 16 processes share the node’s memory bandwidth.Compiler and implementation differences are less visible in the bandwidth-limited multi configuration.
- Coupled Burgers equation: Recording time increases with tape complexity in the single configuration, with Chunk variants requiring around 10% more computation time than Unchecked variants.Index-based variants also store additional data, increasing taping time.
- Coupled Burgers equation: Index tapes use less memory and therefore reduce reverse interpretation time, supporting the conclusion that memory savings can improve AD runtime.The Unchecked and Chunk interpretation times are similar, while index-reuse types are faster.
- Coupled Burgers equation: Burgers derivative computation is 22 times slower than primal execution in the single case and 26 times slower in the multi case.The unusually high factors are associated with the test case containing no transcendental functions.
- Coupled Burgers equation: A default chunk size of 2 million entries balances block-size effects across configurations, while very small and very large blocks can increase interpretation time.For the single case, block sizes up to 0.5 million entries have a substantial influence; 0.5 million entries corresponds to 6 MB for 12-byte entries.
- SU2: SU2 shows no noticeable performance degradation for parallel computation relative to serial computation, indicating that the serial case is already bandwidth limited.The parallel case consistently has a slightly lower factor.
- SU2: In SU2, preaccumulation increases taping time by roughly 40% but reduces interpretation time by almost 50%, overall adjoint runtime by nearly 50%, and memory consumption substantially.Without preaccumulation, recording factors are 3.1 to 4.47 and interpretation factors are 0.4 to 0.86 relative to one primal flow iteration.
- SU2: ChunkIndex does not outperform Chunk in SU2 because numerous active copies make linear indexing sufficient for copy operations.This contrasts with the Burgers case, where index reuse improves reverse interpretation time.
7. CONCLUSION AND OUTLOOK
CoDiPack combines high performance with a modular, extensible design, performing well under compiler, memory-management, and parallel black-box differentiation tests. Future work targets further performance improvements and broader guidance for selecting configurations.
- Conclusion: CoDiPack’s Jacobi taping with expression templates achieves high performance while supporting additional taping approaches and maintaining an easy-to-use interface.Its recursive data layout also permits adding new data streams, while avoiding preprocessor macros improves code understandability.
- Performance evaluation: Intel and gcc produced similar performance, while memory bandwidth reduced compiler differences when all node cores were used.Chunk-based memory management caused only moderate recording degradation and avoided requiring the tape size in advance.
- Application to SU2: CoDiPack retained high performance for black-box differentiation of complex SU2 code in parallel.Preaccumulation can further reduce memory consumption during this application.
- Outlook: Future work will improve performance through novel indexing and memory-management schemes and develop guidance for choosing CoDi types by code structure.The user interface will also be extended with routines for user-differentiated functions and preaccumulation.