PaperlayerView exact revision on arXiv

Source-linked AI summary

Quantum Package 2.0: An Open-Source Determinant-Driven Suite of Programs

Yann Garniron, Thomas Applencourt, Kevin Gasperich, Anouar Benali, Anthony Ferté, Julien Paquier, Barthélémy Pradines, Roland Assaraf, Peter Reinhardt, Julien Toulouse, Pierrette Barbaresco, Nicolas Renon, Grégoire David, Jean-Paul Malrieu, Mickaël Véril, Michel Caffarel, Pierre-François Loos, Emmanuel Giner, Anthony Scemama

arXiv:1902.08154v3physics.comp-phcond-mat.str-elphysics.atom-phphysics.chem-ph

TL;DR

Quantum Package targets scalable wave-function quantum chemistry by combining determinant-driven selected-CI methods with parallel implementations. It introduces renormalized and stochastic perturbative techniques, while routinely handling roughly 2,000 CPU cores and tens of millions of determinants, with tests up to 12,288 cores.

  • Problem

    Quantum chemistry requires methods that can handle large determinant spaces and modern parallel architectures while retaining flexible wave-function descriptions.

  • Method

    Quantum Package implements determinant-driven CIPSI selected CI with PT2, renormalized extrapolation, stochastic selection, and parallel computational algorithms.

  • Results

    The code routinely runs on roughly 2,000 CPU cores with tens of millions of determinants and has been tested up to 12,288 cores; sCI+rPT2 converges faster toward a 7.20 eV excitation energy.

  • Takeaways & Limitations

    The suite supports scalable experimentation with selected-CI wave-function methods and efficient extrapolation toward the full-CI limit.

Abstract

from arXiv · show

\textsc{Quantum Package} is an open-source programming environment for quantum chemistry specially designed for wave function methods. Its main goal is the development of determinant-driven selected configuration interaction (sCI) methods and multi-reference second-order perturbation theory (PT2). The determinant-driven framework allows the programmer to include any arbitrary set of determinants in the reference space, hence providing greater methodological freedoms. The sCI method implemented in \textsc{Quantum Package} is based on the CIPSI (Configuration Interaction using a Perturbative Selection made Iteratively) algorithm which complements the variational sCI energy with a PT2 correction. Additional external plugins have been recently added to perform calculations with multireference coupled cluster theory and range-separated density-functional theory. All the programs are developed with the IRPF90 code generator, which simplifies collaborative work and the development of new features. \textsc{Quantum Package} strives to allow easy implementation and experimentation of new methods, while making parallel computation as simple and efficient as possible on modern supercomputer architectures. Currently, the code enables, routinely, to realize runs on roughly 2\,000 CPU cores, with tens of millions of determinants in the reference space. Moreover, we have been able to push up to 12\,288 cores in order to test its parallel efficiency. In the present manuscript, we also introduce some key new developments: i) a renormalized second-order perturbative correction for efficient extrapolation to the full CI limit, and ii) a stochastic version of the CIPSI selection performed simultaneously to the PT2 calculation at no extra cost.

I. INTRODUCTION

Quantum Package addresses the shift toward memory- and data-movement-limited supercomputing by implementing determinant-driven wave-function methods for scalable parallel computation. Its context combines selected-CI applications with computational bottlenecks in electron-integral transformations and Hamiltonian construction.

  • Motivation: Quantum chemistry algorithms are being redesigned for parallel architectures as data movement increasingly constrains performance.The memory wall makes data movement a central concern, motivating parallel-friendly quantum-chemistry methods.
  • Quantum Package: Quantum Package is an open-source suite designed to simplify implementation, experimentation, and efficient parallel computation of wave-function methods.Its determinant-driven design supports the development of new methods and freely available software.
  • Applications: The package has supported accurate ground- and excited-state calculations for atomic and molecular systems, including more than one hundred transition energies in 18 small molecules.CIPSI wave functions were also used in quantum Monte Carlo studies of water and FeS.
  • Electronic-structure background: Full configuration interaction provides an exact solution within a basis but is usually computationally intractable because its cost scales exponentially with system size.Wave functions are expanded in Slater determinants, whose Hamiltonian matrix elements simplify through Slater–Condon rules.
  • Computational bottlenecks: Post-Hartree–Fock calculations face severe two-electron-integral costs, including O(Norb^8) naive molecular-orbital transformations and difficult distributed parallelization.Density fitting and low-rank approximations are cited as ways to reduce these computational and storage bottlenecks.

C. Selected CI methods

Selected-CI methods choose determinants adaptively according to their estimated contribution rather than imposing occupation-based choices. Quantum Package implements CIPSI with a perturbative correction, retaining methodological flexibility while confronting determinant-driven computational costs.

  • Selected-CI principle: Selected-CI methods select determinants from the full determinant set by estimated contribution to the FCI wave function, avoiding a priori correlation assumptions.This can reduce bias at the cost of a brute-force calculation.
  • CIPSI selection: CIPSI iteratively selects external determinants using a perturbative criterion based on their estimated second-order correlation-energy gain.The selected determinants enlarge the variational reference space across iterations.
  • PT2 correction: The second-order correction estimates the missing correlation energy, approximately E(2) ≈ E_FCI − E(0), for sufficiently large expansions.It complements the variational energy by estimating the remaining correlation contribution.
  • Methodological flexibility: Selected-CI methods can target FCI, complete-active-space, and standard CI determinant spaces by zeroing contributions outside the chosen target space.The same selection framework therefore applies across multiple determinant spaces.
  • Computational trade-offs: Determinant-driven implementations are costly because selected determinants must be explicitly maintained, although compact wave functions make calculations practical.Their determinant count can greatly exceed the formal number of integrals, requiring efficient determinant comparison and integral retrieval.

D. Extrapolation techniques

The paper extrapolates selected-CI energies toward the FCI limit using the second-order Epstein–Nesbet correction and introduces a renormalized PT2 variant. The renormalization dampens corrections for small wave functions without additional computational cost.

  • 1. Usual extrapolation procedure: The extrapolation treats E(2) as an estimate of the selected-CI truncation error and reaches the FCI limit when E(2) = 0.The procedure extrapolates E(0) as a function of E(2).
  • 2. Renormalized PT2: The renormalized correction rPT2 uses an energy-dependent second-order self-energy to modify the usual sCI+PT2 estimate.Its renormalization factor is computed from the same quantities used for E(2).
  • 2. Renormalized PT2: The renormalization factor satisfies 0 ≤ Z ≤ 1 and adds no computational cost when evaluated alongside E(2).The procedure damps E(2) for small reference wave functions, where the unrenormalized correction can overestimate the FCI energy in magnitude.

III. IMPLEMENTATION

Quantum Package’s implementation relies on determinant-driven data structures and algorithms designed for efficient Hamiltonian construction. The implementation discussion covers integral storage, determinant representation, and the organization of core algorithms.

  • Implementation overview: The implementation overview focuses on the data structures and algorithms underlying determinant-driven Hamiltonian matrix construction.The package documents these computational components alongside the implemented methods.
  • Integral storage: Two-electron integrals remain in memory and are stored in a locality-preserving hash table that keeps only nonzero values for fast retrieval.The hash design favors data locality rather than maximal collision avoidance.
  • Integral storage: The hash function maps orbital quartets related by permutation symmetry to keys used for integral lookup.The implementation defines the mapping through successive minimum, maximum, and triangular-index operations.
  • Integral storage: Hash buckets store sorted key fragments, enabling binary search while requiring only two bytes per stored key.Each bucket can hold up to 2^15 consecutive key-value pairs.
  • Determinant representation: Determinants are represented through ordered creation operators and bitstrings encoding spin-orbital occupations.The ordering convention separates spin-up and spin-down orbitals, while phase factors account for fermionic sign changes.

C. Davidson diagonalization

Davidson diagonalization makes determinant-driven CI feasible by targeting only the lowest requested eigenstates, reducing scaling relative to standard diagonalization. The implementation iteratively expands guess vectors with residuals until convergence, while CIPSI builds the reference space through perturbative determinant selection.

  • C. Davidson diagonalization: Davidson targets the Nstates lowest eigenstates and reduces computation and storage to O(NstatesNdet^2) and O(NstatesNdet), respectively.This avoids the prohibitive cost of computing the full spectrum when the determinant space contains millions of determinants.
  • 1. The basic algorithm: CIPSI iteratively adds external determinants to the reference wave function according to a perturbative selection criterion.The implementation improvement focuses on this determinant-selection algorithm.
  • C. Davidson diagonalization: The Davidson procedure orthonormalizes trial vectors, applies H, diagonalizes the projected matrix, and appends residuals until ||R|| < ϵ.The algorithm then returns the converged eigenvectors.
  • 1. The basic algorithm: For each iteration, CIPSI computes external determinant contributions, sums them into the PT2 correction, selects the largest contributions, and repeats until convergence.The variational space is expanded as I_n+1 = I_n ∪ A*_n, with single-state calculations aiming to double its size each iteration.

2. New stochastic selection

The stochastic CIPSI selection removes bias and size-consistency problems associated with thresholded approximations by selecting determinants during the stochastic PT2 calculation. The method preserves the complete wave function, introduces no additional selection cost, and supports balanced multi-state selection and spin-adapted calculations.

  • 2. New stochastic selection: Thresholded CIPSI approximations can bias convergence away from the FCI limit and suffer from size-consistency issues, whereas stochastic selection asymptotically recovers the unbiased FCI solution.The stochastic scheme uses no wave-function threshold, provided the calculation is run long enough.
  • 2. New stochastic selection: The hybrid PT2 computation combines stochastic sampling with deterministic evaluation, yielding an unbiased correction and an exact deterministic value after sufficiently long execution.The stochastic component accelerates convergence and provides a reliable error bar; the default stopping criterion is |δE^(2)/E^(2)| = 0.002.
  • 2. New stochastic selection: Determinant selection is performed on the fly during stochastic PT2 evaluation, completely removing the conventional deterministic selection step at no additional cost.The variational-energy curves versus Ndet are reported as indistinguishable for deterministic and stochastic selection.
  • 2. New stochastic selection: For multiple states, separate PT2 calculations and a modified criterion produce balanced selection across states with different multiconfigurational character.This balances the determinant-selection process rather than favoring states with one particular configuration character.
  • IV. SPIN-ADAPTED WAVE FUNCTIONS: Spin-adapted selection restores S^2 symmetry in the truncated determinant space by adding missing spin-flipped determinants, while the default wave functions increase the internal space by a factor of 2 to 3.The Hamiltonian penalty and shared connected-pair search improve convergence to the desired spin state without extra cost for S^2U.

V. PARALLELISM

Quantum Package combines OpenMP, MPI, and ZeroMQ to parallelize Davidson diagonalization and stochastic PT2/CIPSI selection with elastic resource management. The implementation supports large determinant expansions and applications spanning many molecules and electronic transitions, while remaining limited by single-node memory.

  • V. PARALLELISM: Three parallelism layers—OpenMP, MPI, and ZeroMQ—enable elastic resource management, including dynamically adding jobs or cloud resources during a calculation.The task-based design can start without waiting for all nodes and can combine heterogeneous helpers, including GPU-accelerated and CPU-only programs.
  • V. PARALLELISM: The current limitation is the memory available on a single-node instance, although extending memory across compute nodes is identified as a possible future transformation.The proposed direction uses MPI with scattered data structures.
  • V. PARALLELISM: Davidson diagonalization distributes the W = H U matrix-product tasks, with dynamic scheduling and ZeroMQ task-server communication across helper processes.Each task builds a unique piece of W containing 40 000 consecutive determinants, while U and large constant data are shared within nodes.
  • V. PARALLELISM: Stochastic PT2/CIPSI selection assigns tasks for sampled εJ or εK contributions and associated external determinants, with independent OpenMP-thread clients communicating through ZeroMQ.Common data are broadcast with MPI, and the communication pattern is illustrated for a master node and helper MPI job.
  • A. Capabilities of Quantum Package: 18 small molecules yielded more than 100 highly accurate vertical excitation energies using sCI expansions of several million determinants and augmented triple-ζ basis sets.The study benchmarked 12 excited-state wave-function methods accounting for double and triple excitations.
  • A. Capabilities of Quantum Package: A separate study used more than 200 million determinants for 20 vertical transitions from 14 small- and medium-size molecules, while large sCI expansions addressed three copper complexes.These applications covered substantial double-excitation character and relative energies of low-lying electronic states.

B. Extrapolation

The renormalized PT2 correction improves extrapolation toward the FCI limit for CN3 and Cr2, while Quantum Package supports large-scale parallel calculations and modular method development.

  • B. Extrapolation: 7.20 eV: CN3 sCI+PT2 and sCI+rPT2 excitation energies converge to this value, while rPT2 converges much faster.The value is compatible with the reference excitation energy obtained in a larger basis set, despite E(2) remaining roughly 0.02 a.u.
  • B. Extrapolation: rPT2 makes extrapolation to the FCI limit practically easier by relating E(0) to the renormalized correction Z E(2).Figure 4 compares E(0) against E(2) and Z E(2), including linear fits to the final points.
  • B. Extrapolation: The renormalized correction yields a much straighter extrapolation curve than standard PT2 for strongly correlated Cr2.The authors identify rPT2 as a superior extrapolation framework and report that its renormalization mitigates PT2 overestimation of the FCI limit.
  • C. Speedup: 66% parallel efficiency is obtained for the largest 100 × 10^6-determinant wave function on 192 nodes, or 9216 cores.For the 25 × 10^6-determinant wave function, speedup plateaus at 3072 cores because too few tasks remain to balance the nodes.
  • A. The Quantum Package philosophy: Quantum Package is designed as a standalone, modular library that facilitates new-method development, collaboration, and interfacing with external programs.Its modular organization supports determinant-driven wave-function methods, DFT functionality, and simple import of orbitals or integrals from other programs.

B. The IRPF90 code generator

IRPF90 structures Quantum Package as a provider-based library that simplifies development, modular extension, and large-scale parallel quantum-chemistry calculations.

  • B. The IRPF90 code generator: IRPF90 generates glue code from programmer-written computation kernels, managing dependencies between providers and computed quantities.A production tree represents these dependencies, with each provider computing one entity from valid prerequisite entities.
  • B. The IRPF90 code generator: Quantum Package exposes core wave-function and DFT capabilities through reusable providers, while external modules can be developed and integrated independently.The plugin model lets developers maintain independent repositories and later integrate widely useful plugins into the core.
  • B. The IRPF90 code generator: The second version routinely supports roughly 2,000 CPU cores with tens of millions of determinants and has been tested up to 12,288 cores.The improvements included single-core optimizations and adaptations for large-scale parallelism, particularly load balancing.
  • B. The IRPF90 code generator: New computational developments include a stochastic PT2 correction costing a few percent of deterministic evaluation and CIPSI selection performed alongside PT2 at virtually no extra cost.The stochastic PT2 estimate is unbiased with a controlled statistical error, while stochastic selection is effectively free when PT2 is required.
  • B. The IRPF90 code generator: The modular architecture is designed to let developers create new modules and directly reuse pre-existing Quantum Package work.The code-generator and plugin organization target collaborative development rather than direct modification of the core.

Appendix A: Implementation details

Quantum Package stores two-electron integrals in a locality-aware hash table, accepting moderate overhead while accelerating regular access with a software cache for frequently used orbitals.

  • Appendix A: Implementation details: The access-time benchmark compares array and hash-table storage across random and regular integral-access patterns, excluding random-number generation from random-access timings.The benchmark used a water molecule with the cc-pVQZ basis and 115 molecular orbitals on a dual-socket Intel Xeon processor.
  • Appendix A: Implementation details: Hash-table random access is only 2.18 times slower than array random access, while regular access is roughly three times faster than hash-table random access.The hash table uses data locality and binary search, with two random accesses needed to locate an integral.
  • Appendix A: Implementation details: A CIPSI calculation using hash storage took only twice the wall-clock time of the array-storage calculation.A software cache stores integrals involving the 128 molecular orbitals nearest the Fermi level in a dense 2 GiB array.

2. Internal representation of determinants

Determinants are represented with spin-separated bitstrings, while fixed orbital ordering and phase masks enable efficient excitation and fermionic-sign calculations.

  • 2. Internal representation of determinants: The determinant representation is motivated by fermionic operator ordering, whose permutations introduce the sign changes required for consistent determinant phases.The paper illustrates this with creation-operator permutations and the resulting re-ordered determinant sign.
  • 2. Internal representation of determinants: A fixed ordering places spin-up operators before spin-down operators and sorts each spin sector by increasing orbital index, making the phase factor part of the CI coefficient.Reordering creation operators can change the determinant sign because of fermionic antisymmetry.
  • 2. Internal representation of determinants: Each determinant stores spin-up and spin-down occupations as separate bitstrings, mapping each spinorbital to an occupation bit.This representation uses 64-bit integer arrays, requiring 2 × 64 × Nint bits per determinant.
  • 2. Internal representation of determinants: Low-level bit operations identify the excitation operator connecting two determinants and retrieve the associated two-electron integrals efficiently.The determinant comparison supplies the orbital indices needed for Hamiltonian matrix elements, after which integrals are fetched from the hash table.
  • 2. Internal representation of determinants: A phase mask records the parity of occupied lower spinorbitals, reducing repeated phase-factor evaluation to a few CPU cycles.The mask is built in logarithmic time for groups of 64 molecular orbitals using associative xor operations.

3. Davidson diagonalization

Quantum Package uses direct, determinant-driven Davidson diagonalization that filters for connected determinant pairs and distributes the remaining work across nodes.

  • 3. Davidson diagonalization: Quantum Package implements a multi-state Davidson diagonalization using reordered internal determinants and the Waller-Hartree double-determinant representation.These representations organize the determinant space for the diagonalization procedure.
  • 3. Davidson diagonalization: The implementation computes Hamiltonian elements on the fly because the full matrix may not fit in one node’s memory.Only determinants differing by at most two spinorbitals are connected, but naively examining all determinant pairs scales as N_det^2.
  • 3. Davidson diagonalization: A DISCIUS-like algorithm filters out disconnected determinant pairs so the matrix product iterates only over Hamiltonian-connected pairs.This filtering is necessary for an efficient determinant-driven implementation.
  • 3. Davidson diagonalization: Same-spin single and double excitations can be found in O(N_det^1/2), while opposite-spin double excitations use a two-step search with formal O(N_det^3/2) scaling.Reducing the latter to O(N_det) would require storing all singly and doubly excited determinants, increasing memory usage.

4. CIPSI selection and PT2 energy

Quantum Package computes CIPSI/PT2 contributions by filtering and streaming external determinants, then retaining the most significant perturbative contributions. For CN3, two filtering stages reduce the working determinant expansions substantially.

  • External determinants are generated once by processing generators in decreasing c2 order and excluding internal or previously connected determinants.This streaming construction avoids additional memory for duplicate external determinants.
  • Coarse- and fine-grained filtering exploit excitation and connectivity constraints to avoid scanning the full internal determinant space.The coarse-grained candidate space is bounded by possible quadruple excitations in the two spin sectors rather than scaling as O(Ndet).
  • Filtered internal spaces are used to accumulate external contributions in batches, while banned cells corresponding to internal determinants are excluded from PT2 and selection.The batch contributions are accumulated in a two-dimensional array indexed by the external determinant labels.
  • The method keeps only the Ndet most significant external determinants, sorted by decreasing absolute perturbative contribution.This retains a compact set of determinants for the subsequent selection process.
  • 120 000 determinants typically remain after coarse-grained filtering, an eightfold reduction from the original wave function.The coarse-grained process removes determinants more than quadruply excited relative to each generator.
Loading 1902.08154v3…