Source-linked AI summary
CerebroSim: Scalable Whole-Brain Simulator at 100-Trillion-Synapse Scale on the LineShine Supercomputer
Guangnan Feng, Tianxiang Lyu, Hao Huang, Honghui Liang, Jingjing Li, Zhiguang Chen, Yutong Lu
TL;DR
CerebroSim targets the gap between descriptive brain models and executable simulations, while addressing the communication and memory demands of human-brain-scale spiking models. It combines delay-aware communication, race-free synaptic updates, and compressed or regenerated connectivity. On LineShine, it simulates 86 billion neurons and 100 trillion synapses, sustaining 24.44 PFlop/s with 91% weak-scaling efficiency for the core simulation phase and 94% strong-scaling efficiency for the whole-brain case.
Problem
Descriptive brain models often lack grounded anatomical detail and mechanisms needed for executable simulation, while whole-brain spiking models face irregular communication, multithreaded delivery, and synaptic-memory constraints.
Source: §I · p. 1Method
CerebroSim coordinates delay-aware spike communication, race-free multithreaded synaptic computation, and compressed indexing with on-the-fly synapse regeneration.
Source: §I · pp. 1–2, §VII · p. 10Results
On 18,432 LineShine nodes, CerebroSim simulates 86 billion neurons and 100 trillion synapses at 24.44 PFlop/s, with 91% weak-scaling efficiency for the core simulation phase and 94% strong-scaling efficiency for the whole-brain case.
Source: §VII · p. 10Takeaways & Limitations
CerebroSim provides an infrastructure for biologically constrained spiking models that can support mechanistic studies of brain disorders through controlled in silico perturbations.
Source: §VII · p. 10The gap between fine-grained neural simulation outputs and fMRI signals remains an active challenge, with neurovascular coupling models identified as a needed bridge.
Source: §VII · p. 11
Abstract
from arXiv · showhide
Building executable brain models is essential for moving neuroscience from description to mechanism and prediction. Human-brain-scale spiking simulation is constrained by highly irregular communication, multithreaded spike delivery, and the memory cost of sparse connectivity. We present CerebroSim, a scalable framework for whole-brain simulation. CerebroSim combines Delay-aware Spike Broadcast (DSB) for aggregated delay-aware communication, Race-free Synaptic Dynamics Computation (RSDC) for lock/atomic-free multithreaded delivery with HBM-aware optimization, and Sparse Synapse Storage Compression (3SC) for compact indexing with deterministic synapse regeneration. Using a model derived from magnetic resonance imaging and diffusion-weighted imaging, CerebroSim simulates 86 billion neurons and 100 trillion synapses on 18,432 nodes across 11.2 million cores of the LineShine Supercomputer, sustaining 24.44 PFlop/s, 91% weak-scaling efficiency, and 94% strong-scaling efficiency. This capability makes biologically constrained human-brain models practical for mechanistic studies of brain disorders and controlled in silico testing of intervention hypotheses.
I. INTRODUCTION
CerebroSim addresses the gap between descriptive brain models and executable simulations of neural processes. It coordinates communication, synaptic-update, and connectivity-storage innovations to scale whole-brain spiking simulation.
I. INTRODUCTION
- Descriptive models capture neural-activity correlations but often lack anatomical detail and mechanisms such as spike communication, synaptic plasticity, and emergent computation.The stated goal is executable models that can reproduce, predict, and interact with neural processes.Source: §I · p. 1
- Whole-brain simulation is constrained by scarce high-resolution data, irregular event-driven communication, sparse memory accesses, and the memory cost of synaptic data.These constraints can reduce network utilization and bandwidth efficiency while increasing latency and synchronization overhead.Source: §I · p. 1
- CerebroSim is presented as a unified framework integrating data, models, and high-performance computing for fine-grained neural dynamics at billion-neuron scale.The framework is designed for long-horizon simulations with massive neuronal populations, realistic synaptic delays, and realistic brain architecture.Source: §I · p. 1
- DSB aggregates irregular spike traffic over virtual topologies and uses biological delay windows to schedule transmission and overlap communication with computation.A lightweight transfer semantic is also used to reduce protocol overhead.Source: §I · p. 2
- RSDC uses data partitioning, deterministic scheduling, and conflict-free updates to avoid locks and atomic operations during multithreaded spike delivery.HBM prefetching is combined with the design to mitigate irregular memory accesses.Source: §I · p. 2
- 3SC combines compressed synapse indexing with FlySyn, which regenerates synapses on the fly instead of storing them.Its Synapse-Aware Compressed Index (SACI) compresses index entries.Source: §I · p. 2
A. Current Scale and Performance
Prior work has reached whole-cortex and human-brain scales, while communication and concurrent spike delivery remain key execution challenges. Brain connectivity also makes these workloads highly irregular, with sparse writes that can collide across threads.
A. Current Scale and Performance
- [13] reported a human-brain-scale Digital Brain model with 86 billion neurons and 47.8 trillion synapses on 3,503 nodes and 14,012 GPUs.[12] reported a mouse whole-cortex simulation with 9 million biophysical neurons and 26 billion synapses on Fugaku.Source: A. Current Scale and Performance · p. 2
B. Spike Propagation Communication
- Spike broadcasts combine tiny messages, irregular destinations, heterogeneous connection delays, and sparse firing activity.Existing approaches map this traffic to MPI collectives or rank-to-rank routing, while two-level routing has been used to reduce inter-GPU pressure.Source: B. Spike Propagation Communication · p. 2
- The average 357 B broadcast payload would use about 4.4% of link bandwidth under an 8 KB efficiency target, illustrating the mismatch with hardware and communication-library assumptions.The passage motivates communication methods for tiny payloads, irregular fan-out, and delay-aware propagation.Source: B. Spike Propagation Communication · p. 3
- Skewed firing rates and connectivity can create communication hotspots, and existing MPI collectives are not designed to handle that skew efficiently.Dynamic spike traffic can also limit use of topology-aware routing.Source: B. Spike Propagation Communication · p. 3
C. Race Conditions in Spike Delivery
- Concurrent spike delivery can make multiple threads update the same postsynaptic state, creating races determined by network connectivity rather than thread assignment.Collisions are occasional and scattered in memory because only a small fraction of synapses is active in a step.Source: C. Race Conditions in Spike Delivery · p. 3
- Mutexes or atomic operations, thread-private buffers with reduction, and lock-free structures are existing approaches with different synchronization-cost and memory-overhead trade-offs.These strategies support shared-memory parallelism in spike delivery.Source: C. Race Conditions in Spike Delivery · p. 3
D. Connectivity Storage and Compression
Connectivity storage is a major constraint because even sparse synapse indices create large memory footprints and traffic at brain scale. LineShine offers a hierarchical HBM–DDR memory system and high-bandwidth interconnect for large-scale execution.
D. Connectivity Storage and Compression
- At billions of neurons and trillions of synapses, even a modest spike-synapse index creates substantial memory footprint and traffic.The passage frames data representation and storage reduction as central design choices.Source: D. Connectivity Storage and Compression · p. 3
- Sparse formats store existing synapses and support straightforward traversal, but straightforward storage of connectivity, weights, delays, and state can limit model size to available system memory.Procedural connectivity can eliminate explicit connectivity-storage overhead through deterministic on-the-fly regeneration.Source: D. Connectivity Storage and Compression · p. 3
- Whole-brain simulators face coupled challenges in irregular spike communication, multithreaded-update synchronization and locality, and connectivity-storage memory overhead.These challenges motivate coordinated rather than isolated innovations.Source: D. Connectivity Storage and Compression · p. 3
III. LINESHINE SUPERCOMPUTER
- Each LineShine node has 64 GB of HBM and 512 GB of DDR5, with HBM explicitly manageable for bandwidth-sensitive data and DDR5 for larger-capacity data.The architecture combines high bandwidth with large memory capacity.Source: §III · p. 4
- Compute nodes use the LingQi high-speed interconnect, with eight 200 Gbps network interfaces per node providing an aggregate network bandwidth of 1.6 Tb/s per node.LingQi adopts a 2-plane × 4-rail fat-tree topology, with measured single-hop latency of 1.07 us.Source: §III · p. 4
IV. CEREBROSIM: DESIGN AND INNOVATIONS
CerebroSim translates MRI-derived structural information into a biologically constrained spiking neural network. Its overview includes a delay-aware communication design that aggregates and forwards spike traffic while overlapping communication and computation.
IV. CEREBROSIM: DESIGN AND INNOVATIONS
- The framework translates voxel-level anatomical and connectivity constraints from MRI-derived structural information into neuron- and synapse-level representations.These representations form a biologically grounded spiking neural network.Source: §IV · p. 4
- The communication interface labels separate spike sending by destination ranks and delays from receiving by source ranks and source steps.The overview also labels routing, scheduling, aggregation, and distribution stages.Source: §IV · p. 4
- DSB aggregates tiny spike packets through virtual-topology forwarding and uses biological delays to overlap communication with computation.The figure caption also identifies Verbs send/recv semantics as a way to lower protocol overhead.Source: Figure 3 · p. 4
A. Delay-Aware Spike Broadcast (DSB)
DSB restructures irregular, delay-constrained spike broadcasts through hop-by-hop aggregation and scheduling, while RSDC partitions postsynaptic work to avoid concurrent state writes. HBM placement and software prefetching target the memory-bound access patterns of spike delivery.
A. Delay-Aware Spike Broadcast (DSB)
- DSB routes broadcasts over a reusable virtual topology, aggregates messages by next hop, and forwards them along static routes to reduce tiny-packet injections.The implementation supports HyperX and Dragonfly virtual topologies.Source: A. Delay-Aware Spike Broadcast (DSB) · p. 4
- DSB uses biological delay windows as schedulable slack, sending urgent spikes on time while speculatively transmitting future spikes without changing their application-visible delivery steps.It throttles speculative injection according to unfinished packets and packs spikes into packets up to a fixed maximum size.Source: A. Delay-Aware Spike Broadcast (DSB) · pp. 4–5
- SDT uses Verbs send/recv semantics with pre-posted receives to avoid message-matching overhead, rendezvous handshakes, unexpected-message copies, and peer-specific registered-buffer requirements.An MPI-based fallback supports systems without Verbs.Source: A. Delay-Aware Spike Broadcast (DSB) · p. 5
B. Race-Free Synaptic Dynamics Computation (RSDC)
- RSDC partitions postsynaptic neurons across threads and uses deterministic scheduling and thread-local accumulation so each memory location has at most one writer at a time.A lightweight branch filters irrelevant events without additional data structures or memory overhead.Source: B. Race-Free Synaptic Dynamics… · p. 5, Figure 4 · p. 5
- Spike delivery is memory-bound because sparse synapse-range accesses involve little computation; placing dominant arrays in HBM and prefetching within and across spike iterations targets access latency.Inter-iteration prefetching targets later synapses for the current spike, while inter-loop prefetching targets the first synapse for the next spike.Source: B. Race-Free Synaptic Dynamics… · p. 5
C. Sparse Synapse Storage Compression (3SC)
Sparse inter-voxel connectivity leaves many pre-synaptic neuron IDs without local targets, yet conventional sparse layouts may still store their index offsets. SACI retains entries only for IDs with local postsynaptic targets, reducing this index overhead.
C. Sparse Synapse Storage Compression (3SC)
- Sparse inter-voxel connections leave many pre-synaptic neuron IDs without local targets, but conventional compressed sparse column layouts still store offsets for them.This unnecessary index storage becomes prohibitive at whole-brain scale.Source: C. Sparse Synapse Storage Compression… · p. 6
- SACI retains index entries only for pre-synaptic IDs with postsynaptic targets on the local rank, omitting empty entries.The figure contrasts this layout with adjacency-matrix and compressed-sparse-column storage.Source: C. Sparse Synapse Storage Compression… · p. 6
- By omitting empty index entries, SACI reduces memory usage, especially for whole-brain simulations with sparse inter-voxel connections.Source: C. Sparse Synapse Storage Compression… · p. 6
D. On-the-Fly Synapse Regeneration with FlySyn
FlySyn uses deterministic regeneration of static synapses to replace explicit synapse storage with retained projection data, reducing the asymptotic memory footprint. The paper also packages DSB, SDT, and FlySyn methods as portable libraries with distinct interfaces.
D. On-the-Fly Synapse Regeneration with FlySyn
- FlySyn derives deterministic seeds from neuron, projection, rank, thread, and simulation timing information, then combines them with SACI projection data to regenerate synapse weights, delays, and targets.This permits spikes from the same presynaptic neuron to regenerate the same synapses.Source: D. On-the-Fly Synapse Regeneration… · p. 6
- For static synapses, regeneration reduces rank i's asymptotic memory footprint from O(α_iN + E_i) to O(α_iN), with α_i ≪ 1 at whole-brain scale.Here E_i represents explicit synapse storage, while α_iN captures the SACI; α_i is the fraction of global neurons with targets on rank i.Source: D. On-the-Fly Synapse Regeneration… · p. 6
- FlySyn uses ARM SME outer-product hardware to accelerate bit-shuffling, generating 32-bit random numbers from 8-bit entropy segments for synaptic reconstruction.The generator is designed for portability and specifically optimized for ARMv9 on LineShine.Source: D. On-the-Fly Synapse Regeneration… · pp. 6–7, Figure 6 · p. 6
A. Portable Libraries Design and Implementation
- Three portable libraries isolate DSB, SDT, and FlySyn; their interfaces package broadcast controls, streaming transport, and high- or low-level random-number access.The libraries can be developed and tested independently and migrated to other application scenarios.Source: A. Portable Libraries Design and… · p. 7
B. Whole-Brain Simulation Case
The case study constructs a biologically constrained human-brain spiking model from MRI-derived neuron allocation and diffusion-weighted connectivity. The implementation scales to 86 billion neurons on 18,432 nodes, and the sampled raster plots contrast activity under epilepsy and normal conditions.
B. Whole-Brain Simulation Case
- 86 billion neurons run on 18,432 compute nodes, demonstrating strong and weak scaling for a human-brain-scale spiking model.The model is derived from diffusion tensor imaging data and uses biologically constrained connectivity.Source: B. Whole-Brain Simulation Case · p. 7
- Neuron counts are assigned in proportion to voxel gray-matter volume, while row-normalized diffusion-weighted connectivity gives probabilistic outgoing connections between voxels.The cortex comprises approximately 16 billion neurons and subcortical structures approximately 70 billion.Source: B. Whole-Brain Simulation Case · pp. 7–8
- The model uses leaky integrate-and-fire neurons, postsynaptic-current inputs, and independent Poisson background activity; it omits synaptic plasticity because large-scale whole-brain parameterization lacks sufficient biological evidence.Source: B. Whole-Brain Simulation Case · p. 8
- In plots of 5,638 neurons from four somatosensory-cortex voxels, epilepsy shows more irregular, clustered firing, while normal-condition activity is described as more homogeneous and stable.The plots group neurons by layer and type, with time on the horizontal axis and neurons on the vertical axis.Source: Figure 7 · p. 7, B. Whole-Brain Simulation Case · p. 8
VI. PERFORMANCE EVALUATION
CerebroSim maintains high scaling efficiency from 144 to 18,432 nodes, including for the whole-brain case. The optimization evaluation also reports that DSB hides most communication behind computation and reduces spike-filtering overhead.
A. Scalability
- 91% weak-scaling efficiency is achieved for the core simulation phase at 18,432 nodes, with computation time remaining between 2,867 and 2,930 seconds.Including initialization and finalization lowers end-to-end weak-scaling efficiency to 77%, mainly due to their relatively fixed overhead.Source: A. Scalability · p. 8
B. Effectiveness of the Optimizations
- DSB hides most communication behind computation and reduces spike-filtering overhead compared with MPI collectives.Across 144, 576, and 2,304 nodes, 3D HyperX is consistently the best or near-best virtual topology, while the MPI fallback remains competitive.Source: Figure 9 · p. 9
A. Scalability
- For the whole-brain case, scaling from 4,608 to 18,432 nodes still maintains 94% strong scalability across this specified increase in node count.Source: A. Scalability · p. 9
1) Delay-Aware Spike Broadcast:
The optimization evaluations show that DSB improves application performance over MPI collectives, RSDC reduces spike-delivery costs, and 3SC lowers synaptic memory use through compression and regeneration.
1) Delay-Aware Spike Broadcast:
- 59%–262% end-to-end application performance gains over MPI collectives are reported for DSB.DSB overlaps communication with computation and reduces spike-filtering overhead by targeting broadcasts to data each process needs.Source: 1) Delay-Aware Spike Broadcast · p. 9
- 10%–94% performance improvements over other virtual topologies are reported for 3D HyperX, while the MPI fallback is 19%–35% slower than the best Verbs backend.The 3D HyperX configurations balance virtual path length against per-hop traffic load.Source: 1) Delay-Aware Spike Broadcast · p. 9
2) Race-Free Synaptic Dynamics Computation:
- Optimized multithreaded scheduling consistently reduces total runtime across 144, 288, and 576 nodes.At 144 nodes, the “Other” component falls from 66.1s to 45.9s, a secondary reduction associated with improved load balance and lower synchronization overhead.Source: 2) Race-Free Synaptic Dynamics… · p. 9
- Placing both the spike-synapse index and synapse array in HBM yields the largest speedup, and all tested inter-iteration prefetch configurations improve performance.Stride 1 performs best; larger strides can fetch unnecessary data, while inter-loop prefetching adds a modest speedup.Source: 2) Race-Free Synaptic Dynamics… · p. 9
3) Sparse Synapse Storage Compression:
- 65.0% of baseline memory use remains with SACI, falling to 21.8% with RNG-based on-the-fly regeneration in an evaluation covering about 0.78% of whole-brain regions.The evaluation used 144 nodes; memory use grows nearly linearly with scale because scale changes neurons per voxel but has little impact on the connectivity coefficient α.Source: 3) Sparse Synapse Storage Compression · pp. 9–10, Figure 11 · p. 10
- 112 GOPS per core is the peak throughput of the FlySyn regeneration kernel, with negligible impact on overall execution time.For workloads with intensive random-number requirements, FlySyn is theoretically capable of delivering 1.732 EOPS on LineShine.Source: 3) Sparse Synapse Storage Compression · p. 10
VII. CONCLUSION AND FUTURE WORK
CerebroSim combines communication, memory-access, and connectivity redesign to simulate a whole-brain-scale model at high performance. The authors identify scarce high-resolution neuronal recordings as a continuing challenge while describing applications in brain-disorder studies and other irregular workloads.
VII. CONCLUSION AND FUTURE WORK
- The authors present CerebroSim as infrastructure for mechanistic brain-disorder studies through controlled in silico perturbations, including epilepsy and neurodegeneration.They also describe it as a testbed for neuromorphic computing and a potential platform for virtual drug screening.Source: §VII · p. 10
- The three innovations are presented as a co-design template for brain simulation and other irregular, graphlike workloads, including sparse AI and agent-based modeling.The proposed design combines delay-aware communication, locality-aware memory orchestration, and compressed connectivity with on-the-fly regeneration.Source: §VII · p. 10
- On 18,432 LineShine nodes and 11.2 million cores, CerebroSim simulates 86 billion neurons and 100 trillion synapses at 24.44 PFlop/s, with 91% weak- and 94% strong-scaling efficiency.The weak-scaling figure is for the core simulation phase, and the strong-scaling figure is for the whole-brain case.Source: §VII · p. 10
- High-resolution, large-scale neuronal recordings remain scarce, and no single modality yet captures spike-level activity, structural connectivity, and whole-brain dynamics simultaneously.The authors describe the gap between fine-grained neural simulation and coarser brain observation as an active, tractable frontier as experimental and computational methods co-evolve.Source: §VII · pp. 10–11
BIOGRAPHIES
The biographies describe authors whose research interests span high-performance computing, brain simulation, and related computing systems.
BIOGRAPHIES
- Guangnan Feng’s interests include high-performance and scientific computing, networks, and communication software.He is a postdoctoral researcher at Sun Yat-sen University.Source: BIOGRAPHIES · p. 12
- Tianxiang Lyu, Hao Huang, Honghui Liang, and Jingjing Li list high-performance computing and brain simulation among their interests.Their biographies also mention interests such as numerical analysis, quantum computing, or large-scale simulation.Source: BIOGRAPHIES · p. 12
- Zhiguang Chen’s interests include distributed file systems, network storage, and solid-state storage; Yutong Lu’s include computer architecture and programming models.Lu is also identified as Chief Designer of the LineShine Supercomputer.Source: BIOGRAPHIES · p. 12