PaperlayerView exact revision on arXiv

Source-linked AI summary

Extreme-Scale Linear-Scaling Kohn-Sham DFT at 100 Million Atoms: Bridging Quantum Simulations and Experiments

Qimen Xu, Yu Zhang, Dixing Ni, Lei Gao, Guangnan Feng, Qinrui Zheng, Jianting Liu, Haitian Lu, Zhaopeng Jia, Wei Xue, Shriram Chandran, Torsten Hoefler, Haohuan Fu, Yutong Lu

arXiv:2609.13115v1cs.CEcs.DCphysics.comp-ph

TL;DR

Kohn–Sham DFT's cubic scaling has kept calculations to a few hundred to a few thousand atoms, short of experimentally relevant interfaces. XLSDFT uses a divide-and-conquer density-matrix decomposition and Chebyshev-filtered subspace iteration for linear-scaling DFT. It demonstrates silicon calculations up to 200 million atoms and an 11-million-atom battery interface whose calculated trends agree with XPS.

  • Problem

    Conventional Kohn–Sham DFT is limited to a few hundred to a few thousand atoms, while chemically complex interfaces at experimentally relevant scales lack DFT-level electronic-structure observables.

    Source: §III · pp. 1–2

  • Method

    XLSDFT combines divide-and-conquer decomposition of the one-particle density matrix with Chebyshev-filtered subspace iteration in an O(N) KS-DFT framework.

    Source: §III · p. 2

  • Results

    200-million-atom silicon extends the prior 10-million-atom DFT record by an order of magnitude; 11-million-atom Li/LGPS calculations reproduce XPS interface-to-bulk trends.

    Source: §I · p. 1, A. Large-Scale Ab Initio Simulation:… · p. 3, D. Scientific Results: Li/LGPS Battery… · p. 9

  • Takeaways & Limitations

    The 11-million-atom calculation provides DFT electronic-structure observables at the scale of experimentally characterized interphase layers for comparison with XPS.

    Source: §III · p. 2

    Before XLSDFT, O(N) methods had been demonstrated only on single-element or single-phase systems, not for full observables at metallic multi-element interfaces.

    Source: A. Large-Scale Ab Initio Simulation:… · p. 3

Abstract

from arXiv · show

Kohn-Sham density functional theory (DFT) remains the workhorse of ab initio materials simulation, yet cubic computational and quadratic memory scaling have confined calculations to a few hundred to thousands of atoms, spanning only nanometers, far below experimentally relevant length scales. We introduce XLSDFT, a linear-scaling DFT framework based on divide-and-conquer decomposition of the one-particle density matrix and Chebyshev-filtered subspace iteration, achieving linear computational and memory scaling while retaining DFT accuracy. Deployed on the LineShine exascale supercomputer, XLSDFT reduces computational complexity by orders of magnitude, enabling unprecedented DFT scale: a 200-million-atom silicon crystal, twentyfold beyond the prior record. Our implementation achieves 96.6% weak-scaling efficiency and sustained 157.9 Pflop/s (FP64) for a 100-million-atom scaling study. We further simulate an 11-million-atom all-solid-state battery interface of unprecedented complexity, 1,000 times beyond prior DFT for such systems, revealing how lithium metal reacts with the solid electrolyte at atomic resolution, in quantitative agreement with spectroscopy experiments.

I. CONTRIBUTIONS AND PERFORMANCE HIGHLIGHTS

XLSDFT brings Kohn–Sham DFT to silicon systems of up to 200 million atoms and an 11-million-atom battery interface. These demonstrations address the scale gap between routine DFT and experimentally relevant interfaces, where electronic-structure calculations can be compared with measurements.

I. CONTRIBUTIONS AND PERFORMANCE HIGHLIGHTS

  • 100- and 200-million-atom silicon calculations extend the prior 10-million-atom DFT record by an order of magnitude; the 100-million-atom study sustained 157.9 FP64 Pflop/s with 96.6% weak-scaling efficiency.The 11-million-atom battery interface is also reported as 1,000× beyond prior DFT for such complex systems.Source: §I · p. 1, A. Large-Scale Ab Initio Simulation:… · p. 3

III. OVERVIEW OF THE PROBLEM

  • Conventional DFT is typically limited to nanometer scales, while relevant interfaces extend to tens of nanometers or more, creating a gap for direct quantitative comparison with experiments.The cited overview states that existing methods had not delivered DFT-level observables for chemically complex, multi-element interfaces spanning tens of nanometers to micrometers.Source: §III · pp. 1–2, Figure 1 · p. 2
  • The 11-million-atom Li/LGPS interface is the first DFT simulation at the scale of experimentally characterized interphase layers, with electronic-structure observables validated against XPS.The workflow generates the structure with MLFF-MD, computes observables with XLSDFT, then compares them quantitatively with XPS measurements.Source: §III · p. 2

A. Large-Scale Ab Initio Simulation: Progress and Limitations

  • Prior approaches trade scalability or scope: cubic-scaling codes remain limited in routine practice, while existing linear-scaling methods had not covered metallic multi-element interfaces with a full suite of electronic-structure observables.The cited review notes that some sublinear-scaling performance depended on quasi-2D geometry and that existing linear-scaling demonstrations focused on single-element or single-phase systems.Source: A. Large-Scale Ab Initio Simulation:… · pp. 2–3

B. Machine-Learning Electronic Structure Methods

Machine-learning electronic-structure approaches had not established accuracy for chemically complex interfaces or demonstrated scale sufficient for tens-of-nanometers models. The paper frames XLSDFT as addressing this gap through linear-scaling DFT, while the underlying Kohn–Sham solve is dominated by iterative diagonalization.

B. Machine-Learning Electronic Structure Methods

  • Machine-learning electronic-structure methods had been validated on fewer than a few hundred atoms in simple chemical environments, leaving accuracy for complex multi-element interfaces unestablished.Even accurate density predictions do not provide orbital-resolved observables such as DOS, PDOS, and Bader charges without KS eigenstates.Source: B. Machine-Learning Electronic… · p. 3

C. The Gap This Work Addresses

  • XLSDFT addresses the gap by combining O(N) KS-DFT with exascale-optimized parallelism and hardware-specific kernels to deliver electronic-structure observables at experimentally relevant scales.The implementation uses FP64-dominant arithmetic, with FP32 for AMG preconditioning and FP16 for orbital warmstart storage.Source: C. The Gap This Work Addresses · p. 3

V. INNOVATIONS REALIZED

  • Iterative diagonalization of the KS Hamiltonian to find occupied orbitals costs O(N_d N_s^2), yielding O(N^3) overall cost.The cited passage identifies this as the dominant computational step that XLSDFT's innovations target.Source: §V · p. 4

1) Density Matrix Formulation and Nearsightedness

XLSDFT replaces global eigenpair computation with a density-matrix formulation grounded in nearsightedness. It partitions the domain into buffered subdomains so local density matrices can approximate the global matrix and their observables can be assembled in parallel.

1) Density Matrix Formulation and Nearsightedness

  • The electron density is the diagonal of the density matrix, while the electrostatic potential comes from a Poisson solve that replaces quadratic Hartree-energy summation with O(N) work.Source: 1) Density Matrix Formulation and… · p. 4
  • The density matrix decays exponentially in real space for insulating and metallic systems at finite temperature, even when individual KS orbitals are delocalized.This nearsightedness is the theoretical basis for the divide-and-conquer construction.Source: 1) Density Matrix Formulation and… · p. 4
  • Computing the global density matrix from all Ns ∼ N eigenpairs of H creates an O(N^3) bottleneck that XLSDFT's density-matrix formulation addresses.Ground-state observables are expressed in terms of the one-particle density matrix.Source: 2) Divide-and-Conquer Domain… · p. 4, 1) Density Matrix Formulation and… · p. 4

2) Divide-and-Conquer Domain Partitioning

  • The global domain is divided into non-overlapping core regions with surrounding buffers, and each local Hamiltonian is formed by restricting the effective potential to its subdomain.Each local subsystem density matrix is built from the reduced Hamiltonian's local eigenpairs.Source: 2) Divide-and-Conquer Domain… · pp. 4–5
  • The buffer makes each local density matrix approximate the global one in the core, with convergence improving exponentially as buffer thickness increases.The approximation is stated for x in the core region.Source: 2) Divide-and-Conquer Domain… · p. 5
  • A binary partition of unity restricts each subsystem contribution to its interior, enabling assembly of global observables without explicitly forming subsystem density matrices.The partition is used to derive observable assembly formulas from local eigenpairs.Source: 2) Divide-and-Conquer Domain… · p. 5
  • For a fixed effective potential, subsystem calculations are mutually independent, allowing the local problems to be solved in parallel.Source: 2) Divide-and-Conquer Domain… · p. 5

3) Subsystem Solution via CheFSI

Each subsystem KS problem is solved with Chebyshev-filtered subspace iteration (CheFSI), which avoids full diagonalization while spanning the occupied subspace. With fixed subsystem size, the approach yields linear total cost and memory as the system grows.

3) Subsystem Solution via CheFSI

  • CheFSI avoids explicit full diagonalization of each local Hamiltonian while accurately spanning the occupied subspace, with minimal orthogonalization and memory overhead.Source: 3) Subsystem Solution via CheFSI · p. 5
  • Chebyshev filtering applies a degree-m polynomial through sparse Hamiltonian-vector products, amplifying occupied components and suppressing eigencomponents above the cutoff.Source: 3) Subsystem Solution via CheFSI · p. 5
  • The filtered basis is used for Rayleigh-Ritz projection and a reduced dense generalized eigenproblem, after which subspace rotation updates the orbitals.The resulting orbitals and local eigenvalues form the CheFSI output for observable assembly.Source: 3) Subsystem Solution via CheFSI · p. 5
  • Because subsystem size stays fixed as N grows, each subsystem costs O(1), giving O(N) total cost and O(N) memory.Source: 3) Subsystem Solution via CheFSI · p. 5

4) Assembly of Global Observables

XLSDFT assembles global electronic observables from local eigenpairs and interior-region weights. This yields density, energy, force, DOS, and projected DOS quantities without losing the framework's linear complexity for DOS and PDOS.

4) Assembly of Global Observables

  • The electron density is assembled from local eigenstate weights in the interior partition, with the chemical potential fixed by charge neutrality.Source: 4) Assembly of Global Observables · p. 5
  • The local eigenvalues and orbitals support calculation of band energy and entropy energy, while atomic forces use a Hellmann-Feynman formulation.Source: 4) Assembly of Global Observables · p. 5
  • PDOS is obtained by projecting local KS orbitals onto atom-centered orbitals, with orbital-resolved contributions combined into atom- and species-resolved quantities.Source: 4) Assembly of Global Observables · p. 5
  • DOS is constructed directly from local eigenpairs, and both DOS and PDOS retain O(N) complexity because they reuse available eigenpairs and interior weights or fixed-count orbital projections.Source: 4) Assembly of Global Observables · p. 5

5) Self-Consistent Field Procedure

The SCF procedure assembles potentials, solves independent subsystem problems, determines the chemical potential, and updates and mixes the density. Communication-efficient chemical-potential updates and material-aware density treatment support the extreme-scale battery-interface calculation.

5) Self-Consistent Field Procedure

  • Each SCF iteration constructs the global effective potential, restricts it to subsystem domains, and solves subsystem KS problems independently with CheFSI.Potential construction and restriction cost O(N); aggregate CheFSI work is O(N).Source: 5) Self-Consistent Field Procedure · p. 6
  • Four to five global reductions, rather than approximately 25, cut chemical-potential wall time from approximately 5–6 s to 80 ms.A DOS histogram provides an initial chemical-potential estimate, followed by three to four Newton refinements.Source: 5) Self-Consistent Field Procedure · p. 6
  • After the chemical potential is determined, local eigenpairs assemble the new electron density at O(N) cost without inter-process communication.Source: 5) Self-Consistent Field Procedure · p. 6
  • For the Li/LGPS interface, an improved initial density and material-masked elliptic preconditioner support robust SCF convergence, with metallic screening activated in Li and suppressed in insulating LGPS.Source: 5) Self-Consistent Field Procedure · p. 6
  • Collective communication is concentrated in the O(N) Poisson solve and the four to five reductions for the chemical-potential solver; subsystem CheFSI requires no algorithmic inter-process communication.Source: 5) Self-Consistent Field Procedure · p. 6

B. HPC Implementation on the LineShine system

XLSDFT combines parallelism and architecture-specific kernels with memory-aware implementation choices to support its production runs.

B. HPC Implementation on the LineShine system

  • Custom BLAS/LAPACK implementations are specialized to XLSDFT’s matrix dimensions, layouts, and operation patterns.The general implementation retains vendor-optimized BLAS/LAPACK outside these specialized paths.Source: B. HPC Implementation on the LineShine… · p. 6
  • OpenMP, ARM SVE vectorization, and ARM SME matrix computation target different parts of the workload within each MPI process.SME is used in custom dense linear-algebra kernels for CheFSI projection and rotation.Source: B. HPC Implementation on the LineShine… · p. 6
  • The specialized configuration recovers approximately 1 GB of HBM per NUMA domain and avoids library-managed temporary buffers in DDR.This capacity allows the finite-difference stencil, nonlocal-projector application, Poisson solve, and most associated data to reside in HBM.Source: B. HPC Implementation on the LineShine… · p. 6

1) Structured AMG Preconditioned Poisson Solver

XLSDFT combines a structured AMG-preconditioned Poisson solve with optimized finite-difference and pseudopotential kernels. The optimizations improve stencil throughput and support convergence at extreme scale.

1) Structured AMG Preconditioned Poisson Solver

  • The Poisson system is solved by conjugate gradients with a structured AMG preconditioner, avoiding the several-thousand iterations that unpreconditioned solves can require at tens of millions of atoms.The finest AMG level uses an inexact matrix-free seven-point approximation, while the outer CG solve retains the full 12th-order operator; the mixed-precision V-cycle keeps Poisson-solve time below 3 s per SCF iteration through 200 million atoms.Source: 1) Structured AMG Preconditioned… · p. 6

2) SVE-Optimized Finite-Difference Stencil

  • The stencil is memory-bandwidth-bound at approximately 0.97 flop/byte, and packed 16-band layouts, cache blocking, and SVE vectorization target its data movement.The implementation also uses boundary-aware neighbor reuse, fused Chebyshev recurrence, and FP16 orbital warmstarts that halve HBM traffic for the first filter step while subsequent arithmetic remains FP64.Source: 2) SVE-Optimized Finite-Difference… · p. 7
  • 5.25 Tflop/s per node, up from 1.66 Tflop/s, gives a 3.2× speedup over a plain SVE implementation.On 20,480 nodes, the 100-million-atom calculation reaches 143 PB/s peak aggregate HBM bandwidth for the Laplacian and 107 PB/s for Vnl.Source: 2) SVE-Optimized Finite-Difference… · p. 7

3) Custom Dense Linear Algebra

Custom dense linear algebra accelerates the projection and rotation stages of CheFSI. The reported applications span large silicon crystals and a Li/LGPS interface, running on the LineShine LX2 system.

3) Custom Dense Linear Algebra

  • Projection and subspace rotation use custom FP64 dense kernels, with projection and rotation reaching 1.20 and 1.31 Eflop/s, respectively, for the 100-million-atom calculation.The kernels use SME outer-product accumulation and specialized parallel implementations of the required BLAS/LAPACK operations.Source: 3) Custom Dense Linear Algebra · p. 7

A. Applications Used to Measure Performance

  • The performance applications include a 100-million-atom silicon weak-scaling endpoint, a converged 200-million-atom silicon crystal, and an 11-million-atom Li/LGPS interface.The interface cell spans 57×44×90 nm, was thermally equilibrated at 300 K, and reached the SCF threshold after 167 accumulated iterations.Source: A. Applications Used to Measure… · p. 7

B. HPC System

  • Each LX2 processor combines two compute dies with 304 cores total and eight on-package HBM stacks in one package.The processor also has 32 GB of HBM with 4 TB/s aggregate bandwidth; the LineShine nodes use two LX2 processors each.Source: B. HPC System · pp. 7–8, Figure 4 · p. 8

C. Measurement Methodology

The paper reports timing and operation-count methods alongside weak- and strong-scaling results and sustained performance. XLSDFT reaches high weak-scaling efficiency and sustains 157.9 Pflop/s in the 100-million-atom CheFSI calculation.

C. Measurement Methodology

  • Timing averages exclude the first and last SCF iterations, while time-to-solution includes I/O and sustained-performance calculations exclude it.Sustained FP64 performance is estimated from analytic operation counts for the finite-difference stencil and CheFSI dense-matrix routines.Source: C. Measurement Methodology · p. 8

A. Scalability

  • 96.6% weak parallel efficiency is reached at 20,480 nodes for the 104,857,600-atom silicon endpoint.The endpoint takes 71.2 s per SCF iteration; the weak-scaling study grows atom count and node count twentyfold.Source: A. Scalability · p. 8
  • 81.3% strong-scaling efficiency is achieved across a 20-fold node increase, while total memory scales linearly with atom count.For a fixed 5,242,880-atom silicon crystal, iteration time falls from approximately 68.8 s to approximately 4.2 s.Source: A. Scalability · p. 8

B. Sustained Performance

  • 157.9 Pflop/s in FP64 is sustained during CheFSI at 104,857,600 atoms on 20,480 LineShine nodes.This rate is computed from analytic operation counts and measured CheFSI wall time; I/O is excluded from the performance rates.Source: B. Sustained Performance · p. 8

C. Time-to-Solution

XLSDFT scales to 104,857,600 silicon atoms with 96.6% weak-scaling efficiency, while an 11-million-atom Li/LGPS model resolves interfacial reduction in agreement with XPS. The results connect extreme-scale DFT to experimentally relevant interfaces and potential interface-design targets.

C. Time-to-Solution

  • Table III compares wall time and FP64 performance for the largest XLSDFT calculations on LineShine.Total wall time includes initialization, I/O, and energy evaluation, plus PDOS calculation for Li/LGPS.Source: C. Time-to-Solution · p. 8, Table III · p. 9

D. Scientific Results: Li/LGPS Battery Interface

C. Time-to-Solution

  • 96.6% weak-scaling and 81.3% strong-scaling efficiencies were measured for XLSDFT on silicon; Fig. 5 also reports SCF wall-time breakdowns against ideal scaling.Weak scaling increased atoms and nodes together from 5,242,880 atoms on 1,024 nodes to 104,857,600 atoms on 20,480 nodes.Source: Figure 5 · p. 9

D. Scientific Results: Li/LGPS Battery Interface

  • Occupied-PDOS integrals show low-valence Ge and P enriched near both interfaces, decaying toward bulk-like LGPS with pronounced in-plane heterogeneity.Larger occupied-PDOS integrals indicate lower local valence.Source: D. Scientific Results: Li/LGPS Battery… · p. 9, Figure 6 · p. 10
  • XPS depth profiles independently show interfacial low-valence Ge and P diminishing toward bulk-like LGPS, consistent with the computed interface-to-bulk trend.The calculation additionally resolves in-plane heterogeneity that is difficult to access experimentally.Source: D. Scientific Results: Li/LGPS Battery… · p. 9

VIII. IMPLICATIONS

  • The 100-million-atom scaling study, 200-million-atom silicon demonstration, and 11-million-atom battery-interface calculation establish XLSDFT as a tool for real materials problems.For Li/LGPS, the computed results point to suppressing interfacial reduction and blocking electron leakage as design targets.Source: §VIII · p. 9
Loading 2609.13115v1…