PaperlayerView exact revision on arXiv

Source-linked AI summary

The gem5 Simulator: Version 20.0+

Jason Lowe-Power, Abdul Mutaal Ahmad, Ayaz Akram, Mohammad Alian, Rico Amslinger, Matteo Andreozzi, Adrià Armejach, Nils Asmussen, Brad Beckmann, Srikant Bharadwaj, Gabe Black, Gedare Bloom, Bobby R. Bruce, Daniel Rodrigues Carvalho, Jeronimo Castrillon, Lizhong Chen, Nicolas Derumigny, Stephan Diestelhorst, Wendy Elsasser, Carlos Escuin, Marjan Fariborz, Amin Farmahini-Farahani, Pouya Fotouhi, Ryan Gambord, Jayneel Gandhi, Dibakar Gope, Thomas Grass, Anthony Gutierrez, Bagus Hanindhito, Andreas Hansson, Swapnil Haria, Austin Harris, Timothy Hayes, Adrian Herrera, Matthew Horsnell, Syed Ali Raza Jafri, Radhika Jagtap, Hanhwi Jang, Reiley Jeyapaul, Timothy M. Jones, Matthias Jung, Subash Kannoth, Hamidreza Khaleghzadeh, Yuetsu Kodama, Tushar Krishna, Tommaso Marinelli, Christian Menard, Andrea Mondelli, Miquel Moreto, Tiago Mück, Omar Naji, Krishnendra Nathella, Hoa Nguyen, Nikos Nikoleris, Lena E. Olson, Marc Orr, Binh Pham, Pablo Prieto, Trivikram Reddy, Alec Roelke, Mahyar Samani, Andreas Sandberg, Javier Setoain, Boris Shingarov, Matthew D. Sinclair, Tuan Ta, Rahul Thakur, Giacomo Travaglini, Michael Upton, Nilay Vaish, Ilias Vougioukas, William Wang, Zhengrong Wang, Norbert Wehn, Christian Weis, David A. Wood, Hongil Yoon, Éder F. Zulian

arXiv:2007.03152v2cs.AR

TL;DR

Computer architecture research needs an open, configurable simulator that can model modern systems with sufficient timing fidelity and broad community support. This paper surveys gem5’s usage, architecture, development, and governance, documenting its evolution and future direction. gem5 is positioned to continue maturing through validated configurations, improved support, and community development.

  • Problem

    Computer architecture research requires simulation infrastructure that models modern hardware with useful timing fidelity while supporting diverse systems and architectures.

  • Method

    The paper surveys gem5’s usage and features, describes its current architecture and development practices, and enumerates major changes since its initial release.

  • Results

    gem5 has become an increasingly important computer architecture research infrastructure with continued additions of features and models and improvements in simulator stability.

  • Takeaways & Limitations

    Known-good configurations, reproducible-simulation tools, documentation, and learning materials are intended to expand gem5’s user base and support continued development.

  • Takeaways & Limitations

    gem5’s SystemC implementation may not support code relying on nonstandard or implementation-specific reference-kernel behavior, despite passing almost all reference tests.

Abstract

from arXiv · show

The open-source and community-supported gem5 simulator is one of the most popular tools for computer architecture research. This simulation infrastructure allows researchers to model modern computer hardware at the cycle level, and it has enough fidelity to boot unmodified Linux-based operating systems and run full applications for multiple architectures including x86, Arm, and RISC-V. The gem5 simulator has been under active development over the last nine years since the original gem5 release. In this time, there have been over 7500 commits to the codebase from over 250 unique contributors which have improved the simulator by adding new features, fixing bugs, and increasing the code quality. In this paper, we give and overview of gem5's usage and features, describe the current state of the gem5 simulator, and enumerate the major changes since the initial release of gem5. We also discuss how the gem5 simulator has transitioned to a formal governance model to enable continued improvement and community support for the next 20 years of computer architecture research.

1 THE GEM5 SIMULATOR

gem5 is an open, community-supported, cycle-level simulator for configurable computer-system research. Since 2011, its expanding community has improved the infrastructure through new models, development practices, governance, and support for reproducible research.

  • Since its 2011 release, gem5 development has accelerated as its contributor community has grown.The paper presents yearly commits and unique contributors as evidence of this acceleration.
  • The project responded to community growth with formal governance, improved documentation, distributed development tools, and broader outreach.Git, Gerrit, continuous integration, and issue tracking made contributions easier and helped improve stability; more than 250 issues were closed in the first six months of bug tracking.
  • Future development emphasizes stable APIs, known-good and publicly validated configurations, better testing, and learning materials to support reproducible simulation and broaden gem5’s user base.These efforts aim to reduce baseline-configuration work while supporting continued community growth.
  • Its Python-based scripting interface dynamically composes modular models, allowing researchers to configure complex systems and select hardware parameters.The simulator includes over 300 parameterized models and a clean model API for extension.
  • gem5’s execute-in-execute design functionally completes operations when timing simulation reaches them, preserving behaviors that depend on thread or I/O timing.This contrasts with trace-based and execute-ahead approaches that may hide timing-dependent behaviors.
  • gem5 provides parameterized models for CPUs, DRAM, interconnects, caches, devices, and other system components in an event-driven, cycle-level simulation environment.Its models can be customized for different systems and span multiple fidelity-performance trade-offs.

2 MAJOR CHANGES IN GEM5-20

gem5-20 consolidates major improvements to resources, documentation, ISA support, and full-system capabilities. The release also expands RISC-V support and addresses the project’s steep learning curve.

  • 7015 commits from at least 250 contributors produced a comprehensive set of gem5 features, bug fixes, and code improvements since 2011.The listed changes are major but not exhaustive.
  • 2.1.1 Testing gem5-20 with gem5 Resources: Shared gem5 resources provide operating-system images, benchmarks, kernels, and build scripts to improve experiment reproducibility.Researchers can start from common artifacts and modify the included sources and scripts.
  • 2.1.1 Testing gem5-20 with gem5 Resources: The resources effort supports testing across SPEC 2006, SPEC 2017, PARSEC, NAS Parallel Benchmarks, GAPBS, and multiple long-term-support Linux kernels.
  • 2.2 Learning gem5: Learning gem5 addresses the simulator’s steep learning curve through step-by-step coding examples covering getting started, extension, and Ruby cache coherence.The material uses a bottom-up approach from simple designs to more realistic examples.
  • 2.3 RISC-V ISA Support: RISC-V support enables detailed architectural analysis that alternatives such as Spike and QEMU cannot provide, while avoiding the time cost of RTL simulation.The implementation supports most RISC-V Linux programs in system call emulation mode.
  • 2.3 RISC-V ISA Support: Full-system RISC-V work adds Sv39 paging, a three-level page-table walker, and a TLB, although some steps toward running Linux remain incomplete.

2.4 Arm Improvements

Arm improvements broaden gem5’s architectural and platform coverage across Armv8-A states, SVE, and Trusted Firmware. The new Foundation platform provides compatibility with FVP Foundation software and boot flows.

  • Armv8-A support: gem5 supports Armv8-A AArch32 and AArch64 instruction sets, their interworking, and selected system-level extensions through Armv8.3-A.
  • Scalable Vector Extension: SVE support uses separated register storage and access abstractions to handle variable-length vectors and predicate registers.The design supports vector registers from 128 to 2048 bits in 128-bit increments.
  • Scalable Vector Extension: The SVE design is flexible enough to support vector instruction sets with arbitrarily large vector registers.
  • Trusted Firmware Support: The VExpress_GEM5_Foundation platform is compatible with FVP Foundation, allowing unmodified Foundation software such as TF-A and supporting UEFI-based boot flows.This moves gem5 closer to supporting UEFI-compatible operating systems.

2.5 X86 ISA Improvements27

x86 improvements increase modeling fidelity and execution flexibility through better out-of-order handling, TSO support, modular branch prediction, virtualization, and elastic traces. These changes span both detailed accuracy and faster exploration.

  • Out-of-order execution: Multiple x86 flag registers prevent unnecessary serialization and unlock more instruction-level parallelism in out-of-order execution.
  • Memory consistency: gem5 implements x86 Total Store Order and also supports more relaxed consistency models for RISC-V and Arm in its out-of-order CPU model.
  • Branch prediction: Five branch prediction techniques, including TAGE, are available, and modular predictors can be extended with loop and indirect predictors.TAGE history-register and table sizes can be varied for performance studies.
  • Virtualized fast forwarding: Hardware virtualization supports fast-forwarding and runtime simulator cloning, enabling parallel simulation of different system parameters and cache-warming outcomes.The Python fork call drains the simulator, then uses UNIX fork to create a consistent copy.
  • Elastic traces: Elastic traces achieve 6-8 times the speed of detailed out-of-order simulation while maintaining 83–93% simulation accuracy.The trace player honors dependencies and adapts execution time when the memory system changes.

2.9 Off-Chip Memory System Models31

gem5’s off-chip memory models support diverse memory technologies, QoS policies, power analysis, and heterogeneous DRAM/NVM configurations. Recent changes add LPDDR5 support and integrate extensible mechanisms for evaluating timing, arbitration, and energy behavior.

  • Memory Controller and DRAM Models: gem5 models off-chip memory timing through a memory controller and media-specific interface that defines architecture, timing, and operations.The interface models operations such as activation, precharge, refresh, and low-power modes, and can be modified by extending a Python class.
  • LPDDR5 Support: LPDDR5 configurations add new timing parameters, multi-cycle commands, and interleaved bursts to evaluate emerging memory tradeoffs.These changes require additional checks and optimizations to preserve model integrity when comparing against real hardware.
  • QoS-Aware Memory Control: QoS-aware memory control supports fixed-priority and proportional-fair policies while separating arbitration policy from DRAM timing.Users can implement and plug in their own arbitration policy into the memory controller model.
  • Power and Energy Modeling: gem5 integrates DRAMPower as a library for online power estimation and models DRAM power-down modes in full-system simulations.The integrated model was evaluated with real HPC workloads for power-down analysis.
  • Future Memory Interfaces: A refactored memory interface framework supports configurable DRAM and NVM address ranges within one controller for heterogeneous memory topologies.The approach separates the DRAM interface from the controller and introduces an agnostic NVM interface for emerging memories.

2.10 Classic Caches Improvements38

Classic cache improvements extend gem5’s ability to model coherent hierarchies, cache maintenance, streaming writes, and scalable coherence tracking. The additions include configurable cache behaviors, write-stream optimizations, arbitrary-hierarchy maintenance operations, and distributed snoop filtering.

  • Coherence Infrastructure: The classic memory system implements a configurable snooping MOESI-like protocol across coherent cache hierarchies.Cache, CoherentXBar, and SnoopFilter components implement coherence and reduce unnecessary traffic.
  • Write Streaming: Write-stream detection enables write coalescing and, at a second threshold, cache-allocation bypassing.The mechanism analyzes sequential incoming writes and activates the optimizations at configurable thresholds.
  • Cache Maintenance: Cache maintenance operations can clean, invalidate, or both across arbitrary cache hierarchies relative to a configurable system point.Cleaning writes back dirty copies, while invalidation removes all copies of a cache line.
  • Snoop Filtering: A distributed snoop filter reduces coherence messages from O(N^2) to O(N) with N concurrent requestors.The model also represents limited filter capacity through backinvalidations when the filter becomes full.

2.11 Cache Replacement Policies and New Compression Support39

gem5 broadens cache-model flexibility through reusable replacement policies, compression support, richer Ruby protocols, checkpointing, and improved Arm multicore coherence. Ruby increasingly shares infrastructure with classic caches, while a unified model remains a future goal.

  • Replacement Policies: gem5 supports 13 reusable replacement policies, including LRU, FIFO, Pseudo-LRU, and RRIP variants, for table-like hardware structures.The policies can be paired with multiple hardware units, with planned extensions beyond data caches to structures such as TLBs.
  • Compression Support: Cache compression support includes BDI, C-Pack, and FPCD algorithms with a compression-oriented organization for co-allocating adjacent blocks.Adjacent blocks share a tag entry, but co-allocation requires each block to compress to a specified fraction of a cache line.
  • Ruby Integration: Ruby provides 12 coherence protocols through SLICC, including GPU-specific, region-coherence, token-coherence, and teaching protocols.Ruby now shares replacement protocols, ports, and DRAM controller models with the classic cache system.
  • Future Direction: A unified cache model combining classic-cache composability and speed with SLICC flexibility and fidelity remains a future objective.The current direction is further unification of Ruby and classic cache models rather than a completed unified implementation.
  • Checkpointing: Ruby supports warm-cache checkpoint restoration, enabling simulations to begin from regions of interest without restarting from an empty state.Functional memory-system access without timed events allows higher simulation speeds during checkpoint operations.
  • Arm Support: Arm Ruby support now uses MOESI_CMP_directory by default and fixes protocol deadlocks, functional accesses, DMA bugs, and latency modeling issues.Load-linked/store-conditional operations were also added to MESI_Three_Level for Arm use.

2.14 GPU Compute Model43

gem5’s GPU compute model represents AMD GCN3-based GPU execution with configurable SIMD resources, a generic API, and a staged CU pipeline. It also models kernel dispatch and adds randomized testing for GPU memory systems with weaker consistency models.

  • GPU Architecture: gem5 integrates a compute-based GPU model based on AMD GCN architecture and currently supporting the GCN3 ISA.The model is intended to support modern GPU architectures that follow a SIMD execution model.
  • Execution Model: GPU execution uses wavefronts of SIMD lanes executing instructions in lock-step, with work-groups mapped to individual compute units.Multiple wavefronts form a work-group, and multiple work-groups form a kernel.
  • Compute Units: Compute units are configurable in SIMD-unit count and width and may include scalar units, scratchpad memory, and global memory resources.The current model supports in-order issue of SIMD and scalar instructions.
  • Pipeline: The modeled compute-unit pipeline has fetch, scoreboard check, schedule, and execute stages.The scoreboard stage handles intra-wavefront dependences before instructions are scheduled.
  • Kernel Launch: Kernel dispatch tracks resource availability and requires enough CU resources for all wavefronts in a work-group.The dispatcher manages both in-flight and waiting work-groups and reserves CU resources before dispatch.
  • GPU Memory Validation: An autonomous tester validates GPU memory systems by injecting randomized data-race-free reads and writes synchronized with atomics and memory fences.This approach addresses GPU systems whose weaker consistency models and software-directed coherence differ from CPU-oriented testing assumptions.

2.15 Runtime Power Modeling and DVFS Support46

gem5 added power-aware modeling and distributed simulation capabilities to support efficient system analysis across modern workloads and multi-node systems.

  • Runtime Power Modeling and DVFS Support: gem5 models voltage and frequency domains, enables DVFS control, and aggregates activity-based power consumption over time.The framework supports operating-system governors, autonomous control, and power modeling for SVE vector-unit activity.
  • Timing-agnostic Models: Timing-agnostic VirtIO models provide limited memory-system interactions without timing for block devices, pass-through filesystems, and serial ports.A NoMali stub exposes Arm Mali register interfaces, allowing production GPU drivers to run without simulating the GPU.
  • Distributed System Modeling: dist-gem5 distributes hierarchical cluster simulation across multiple gem5 processes, forwarding packets over TCP and synchronizing them deterministically.Quantum-based synchronization coordinates compute and network nodes while receiver threads handle incoming TCP traffic.
  • Distributed System Modeling: Distributed checkpointing captures external inter-process state, including in-flight network packets, at periodic global synchronization events.This avoids taking checkpoints while messages remain in flight between gem5 processes.

2.18 SystemC Integration

gem5 integrates with SystemC TLM through compatibility layers and native support, allowing researchers to combine gem5 models with SystemC hardware models while preserving modifiability.

  • SystemC Integration: SystemC TLM offers an ecosystem of hardware models, while gem5 supplies modifiable CPU models; compatibility enables combined systems.The integration addresses the lack of accurate and modifiable CPU models in SystemC.
  • gem5 to SystemC Bridge: gem5 can run within a SystemC simulation by replacing its discrete-event kernel with a SystemC-managed process and translating protocols through transactors.The documented transaction overhead is about 8%.
  • Native SystemC Support: gem5 also provides a built-in SystemC kernel and TLM implementation for models recompiled against gem5’s SystemC headers.These models can use gem5 configuration, Python control, and modified bridges to connect TLM sockets to native gem5 ports.
  • Limitations: gem5’s SystemC implementation is based on the standard but may not support code relying on nonstandard reference-implementation behavior.The implementation passes almost all tests from the reference implementation’s test suite.

2.20 Testing in gem553

gem5 combines system-level and unit testing with pre-merge validation and broader periodic tests to improve stability and limit regressions in a large, frequently changing codebase.

  • Testing Infrastructure: TestLib runs compiled gem5 instances on specified architecture simulations and verifies their intended behavior.Its extensible design allows tests to be added as needed, but it is not well suited to individual components.
  • Testing Infrastructure: Unit tests check gem5 classes and functions at finer granularity, helping developers identify broken functionality more quickly.Unit-test coverage is improving and is expected to contribute more to stability in future releases.
  • Continuous Validation: Gerrit runs compilation and test suites for key ISA targets on every submitted patch before merging.The pre-merge suite covers all unit tests and a subset of TestLib tests across major components.
  • Continuous Validation: Larger tests, including full-system boot tests, run regularly but were still manually triggered.The project planned nightly automation and automatic error reports for these longer tests.

2.21 Internal gem5 Improvements and Features

gem5 improved its internal infrastructure through stable APIs, hierarchical statistics, HDF5 output, modern Python bindings, and coroutine support for asynchronous models.

  • Internal Infrastructure: Stable APIs for simulator-core interfaces are intended to support long-term integration with other simulators and projects.The targeted interfaces include event-driven execution and statistics.
  • Statistics and HDF5: The new statistics API organizes statistics into groups matching the SimObject hierarchy and supports subtree-specific dumps.The design also reduces boilerplate and errors when defining statistics.
  • Statistics and HDF5: HDF5 support stores statistics in a binary, hierarchical format with language bindings and efficient time-series storage.Its data is represented internally as N-dimensional matrices, including a time dimension.
  • Statistics and HDF5: HDF5 introduces storage overhead for filesystem-like structures, but incremental sample storage is orders of magnitude smaller than text output.The overhead is quickly amortized when statistics are sampled.
  • Python Interface: gem5 migrated from SWIG to PyBind11 and completed Python 3 compatibility to improve maintainability and future platform support.The migration also enables under-the-hood improvements such as type annotations.
  • Asynchronous Modeling: Coroutines let asynchronous models checkpoint DMA tasks and resume them after device responses arrive.gem5’s coroutine library is currently used by the SMMUv3 and GICv3 ITS models.

2.22 Updating Guest↔Simulator APIs57

gem5 is improving Guest↔Simulator interaction through more portable signaling and automated handling of guest function calls. The new templates marshal guest arguments into gem5 functions and can optionally return values to the guest.

  • Guest applications and gem5 can call each other for non-architectural actions such as skipping uninteresting functions or adjusting guest behavior.Examples include bypassing memory-clearing routines, CPU-speed measurement loops, or delays.
  • Program-counter callbacks let gem5 detect selected guest functions, with improved support for KVM and Arm Fast Model CPU models.Symbols loaded from an operating-system kernel can identify functions before they execute.
  • PC-based signaling is not universal because KVM executes instructions as real hardware would, requiring alternative APIs.The limitation arises because guest requests cannot always be associated with predictable program counters.
  • Historically, guest calling-convention handling was redundant, inconsistent, incomplete, and difficult to maintain.
  • New templates automatically decompose guest function signatures, marshal arguments, invoke arbitrary gem5 functions, and optionally return values to the guest.The mechanism supports either overriding or observing guest behavior after the gem5 function completes.
  • The Figure 6 example shows a guest call using AAPCS32 being detected and dispatched to gem5 function foo() with guest-provided arguments.The accompanying gem5 code illustrates the corresponding invocation after call detection.

3 CONCLUSION

After nine years, gem5 has become an increasingly important community tool for computer architecture research. Future development focuses on reproducible simulation, broader user support, and continued growth over the next 20 years.

  • gem5 has become an increasingly important tool in the computer architecture research community over the past nine years.
  • Future development will add features and models while increasing simulator stability.
  • The overarching future goal is to increase gem5’s user base within computer architecture and across other computer systems research fields.
  • Known-good configurations, reproducible-simulation tools, improved documentation, and learning materials are intended to broaden gem5 use and support.These efforts are planned to continue growing and improving the infrastructure through the next 20 years of computer system development.
Loading 2007.03152v2…