A team of researchers from the Jülich Supercomputing Centre, KTH Royal Institute of Technology, the University of Bonn and the University of Groningen, led by Carlos Falquez, Shiting Long, Nam Ho, Estela Suarez and Dirk Pleiter, now propose a surprising answer: use a processor simulator not just to design chips, but as a full performance engineering workbench. In a study published in Frontiers in High Performance Computing, they enhanced the widely used open-source simulator gem5 with a performance analysis framework that extends the available performance counters, tracks the temporal behaviour of running applications, and links those statistics automatically to specific code regions. The result is a set of performance profiles that can identify code regions with optimisation potential, monitoring quantities that are usually not directly accessible on real hardware.

The motivation is rooted in the sheer complexity of modern processors. Even a single processor core is a system of interacting subsystems, from the micro-architecture of the front-end pipeline, where various factors influence how efficiently functional units are utilised, to multi-level cache hierarchies and memory prefetchers whose combined behaviour of data transport is notoriously difficult to extrapolate from the behaviour of individual components. On real hardware, Performance Management Units (PMUs) can be programmed at run time to count events such as cache misses, but the set of trackable events is limited, cannot be changed once a processor design is final, and is highly vendor-specific. Even when advanced monitoring features such as Intel’s Processor Event Based Sampling, AMD’s Instruction-Based Sampling or Arm’s Statistical Profiling Extension are available, the context for a particular processor behaviour often cannot be recorded, making root cause analysis a matter of expert guesswork.

Simulators offer an alternative, but they come with a fundamental trade-off. Because cycle-accurate simulation runs many orders of magnitude slower than real hardware, in practice it is only affordable when restricting the scope to a single or a small number of processor cores. gem5, which supports several instruction set architectures including x86, Arm and RISC-V, has become a popular choice and is continuously updated to follow new architectural developments. It has already served as a co-design tool in the development of the A64FX processor and in the European Processor Initiative’s design space explorations. Yet gem5 has been notoriously difficult for computational scientists to use: it was built by and for processor architects, requires significant training, and lacks tools to correlate simulation statistics with application code. Even when an application is instrumented to restrict profiling to a region of interest, gem5’s counters are accumulated over the entire region and typically reported as a single aggregate value, hiding the multiple execution phases with very different performance characteristics that may exist within it.

The new framework tackles this with two key additions: time-binned event counters and automatic labelling of executed code regions. The execution timeline of a region of interest is partitioned into fixed-length intervals, each of duration 5000 cycles, and events such as the commit of an instruction are aggregated per bin, producing a sequence of event counts that provides much finer temporal resolution. To reduce stochastic noise in the plots, a median filter from the scipy Python package is applied, suppressing outliers while preserving sharp changes. For code region detection, the instructions of the executable are grouped according to function entry point labels parsed from the objdump tool of GNU Binutils. For each time bin, the five most frequently occurring events are determined, and a function label is associated with the bin when at least five of those instructions belong to that function. The resulting label sequence is converted into labelled time ranges that can be mapped directly onto the time-binned event counters, revealing which routines were executing during each phase.

The framework also implements algorithms to detect specific classes of performance bottlenecks. One concerns sensitivity to instruction data dependencies: in out-of-order processors, fetched instructions wait in an issue queue until their operands are available, and when an application is ported to an architecture with deeper execution pipelines, the probability of instructions waiting for data dependencies increases while pressure on the execution units decreases, causing unexpectedly low performance. The team added event counters that can differentiate between instruction classes and track both the average number of instructions waiting in the issue queue for dependencies to resolve and the average number of instructions stalled due to busy execution units. A second class concerns memory traffic and the average filling of the queue for pending requests at the memory controller, observables that are difficult or impossible to extract reliably from real hardware. A third concerns long-latency instructions, such as conditioned load instructions whose latency cannot be hidden by out-of-order execution; the simulator detects these by measuring the instruction squashing rate, that is, the rate at which speculatively executed instructions must be discarded because of misspeculation.

Three case studies demonstrate the approach across a broad range of computational science. The first is GROMACS, a very widely used molecular dynamics package, whose most expensive kernel computes non-bonded interactions. Using a recent version of GROMACS with a solvation free energy benchmark, the simulations modelled a single Arm Neoverse V2 core supporting SVE instructions, the same core used by NVIDIA in its Grace processor. The profiles showed that the main numerical kernel achieves a throughput of around 1.7 SIMD floating-point instructions per cycle, almost fully utilising the two available SVE pipelines. Crucially, the new counters revealed that for effective execution latencies of four cycles or more, the vectorised kernel becomes sensitive to data dependencies: the average number of stalled instructions drops below one, signalling a decrease in the rate of dependency resolution, while the number of instructions waiting in the issue queue for missing dependencies remains large, verifying that the front-end keeps sending instructions at sufficient rate.

The second case study concerns a kernel from the Lattice QCD solver DD-αAMG, representative of memory-bandwidth-limited stencil applications. The team investigated a recently extended version of the solver that supports solving linear equations with multiple right-hand sides, an optimisation that increases arithmetic intensity but had not delivered the expected performance improvements on real hardware. Using the A64FX processor on the Ookami system, they had identified that effective memory bandwidth varied across blocking sizes, but comprehensive information from cache and memory controllers is almost impossible to extract from real hardware. The gem5 simulations of a read-intensive inner-product kernel showed that larger blocking sizes enhance performance, mirroring real-hardware trends, but also revealed an unexpected performance drop at a blocking size of eight, where the compiler failed to vectorise the code and generated scalar load instructions instead, halving the rate of read requests.

The third case study targeted the breadth-first search algorithm implemented in the Ligra graph framework, whose irregular control flow and memory access patterns stress the branch predictor. The profiles showed poor execution unit utilisation, with an average commit rate below one instruction per cycle most of the time, and a high rate of squashed instructions. The team traced most of the branch mispredictions to the EdgeMapDense routine, where three specific lines each caused about a third of the mispredictions. By moving a conditioned load instruction outside the branch so it could be initiated earlier, a trade-off accepting some unnecessary data transfers, they benchmarked both implementations on an AWS Graviton 4 bare-metal node with Kronecker graphs of scale 21 to 24 and measured a speed-up of around five percent for all problem sizes, confirming that a simulator-detected bottleneck could be fixed on real hardware.

The authors are careful about the limitations. Single-core simulations run about five to six orders of magnitude slower than execution on real hardware, so simulating many-core architectures or full applications is likely to be prohibitively expensive, although distributed frameworks such as COSSIM may help. Optimising code for specific processors also depends on the availability of validated gem5 configurations, and no library of ready-to-use configurations exists yet; previous validation studies have reported average deviations of about ten percent for simpler Arm cores, with discrepancies of up to forty percent in some cases for complex workloads, and there is a risk of mismatches between a specific processor solution and its gem5 model that only show up under particular circumstances. Still, the benefit of the approach is that gem5 already provides a large and diverse set of statistics that can illuminate the often difficult-to-uncover interplay between observed application performance and the complex hardware on which it executes, with much freedom to add further event monitors and analysis mechanisms. The team plans to extend the list of performance issues for which automatic detection can be integrated into gem5, and to conduct further case studies exploring the benefits and disadvantages of the approach. The modified simulator has been made openly available, potentially also benefiting compiler engineers and system software developers far beyond HPC.

Subject of Research: High Performance Computing

Article Title: Processor simulation as a tool for performance engineering

Article References: Falquez, C., Long, S., Ho, N., Suarez, E., & Pleiter, D. (2025). Processor simulation as a tool for performance engineering. Frontiers in High Performance Computing, 3, Article 1669101. https://doi.org/10.3389/fhpcp.2025.1669101

Image Credits: AI Generated

DOI: 10.3389/fhpcp.2025.1669101

Keywords: processor simulation, gem5, performance engineering, high-performance computing, Arm SVE, RISC-V, GROMACS, Lattice QCD, breadth-first search, performance counters, Processor, simulation