Extra-P Turns Scalability Modeling Into a Few Small Experiments

Supercomputers are built to solve enormous problems quickly, yet the applications that run on them routinely fail to use their full power. A code whose runtime grows quadratically with input size will exhaust even the largest machine available, and a weak-scaling experiment whose execution time keeps climbing as nodes are added delivers diminishing returns from parallelism. Such scaling bottlenecks often remain invisible until developers attempt the critical problem or system size, at which point an expensive supercomputer sits poorly utilized while a remedy—sometimes requiring profound algorithmic research—takes time and expertise not every team can afford. The lesson drawn by researchers at ETH Zurich and TU Darmstadt is that these bottlenecks should be caught as early as possible, ideally during the initial phases of development.

Performance models offer a way to see scaling behavior before it bites. A model is a formula expressing a critical metric such as runtime as a function of execution parameters like the degree of parallelism or input size. The function t = c1 + c2·log(p), for example, characterizes an application whose execution time grows logarithmically on top of a constant base as processors are added—a benign weak-scaling scenario. Deviations of actual performance from what the model predicts can point to unforeseen hardware limitations or errors in the model itself. Crucially, the main goal of such a model is not to predict exact runtime at a different scale, although that is often used as a test; the goal is to identify performance bottlenecks, both relative to other parts of the code and compared to developer expectations.

Models can be created analytically or empirically. Analytical modeling means reasoning about the properties of the underlying algorithms, which is challenging for complex applications with thousands of source lines. Analysts usually restrict themselves to a few selected kernels they believe will dominate performance at scale, a practice that risks overlooking bottlenecks because recognizing dominance at scale is non-trivial—strictly speaking, identifying the right kernels requires the very model one is trying to build. Even modeling a selected kernel is error-prone and sometimes impossible, since the performance behavior of library calls often remains opaque, and reasoning alone can hardly determine constants and coefficients. Empirical performance modeling, the process of learning models from performance data, offers a convenient alternative because it can be largely automated and does not require intimate knowledge of the source code.

This is the premise of Extra-P, a project launched around 2012 to democratize performance modeling via automation. When the effort began, tools for measuring and analyzing performance were widely available, but modeling was confined to a community of experts. Extra-P, built entirely in Python for Linux, Windows, and macOS, generates models from a small number of carefully selected experiments—as few as five. The tool consists of six core components organized around a central experiment object holding parameters, call tree, measurements, and models. File readers load data from formats including the CUBE profiles produced by Score-P and JSON lines; modelers translate measurements into models; experiments can be saved as .extra-p files for reuse; and users interact through a command-line interface for scripting on clusters or a Qt-based graphical user interface for interactive exploration of models across the call tree, complete with severity indicators, line and contour plots, surface plots, and heatmaps.

The mathematical core is the Performance Model Normal Form, or PMNF. It rests on two insights: the function classes commonly found to represent communication and computation times as functions of execution parameters are limited to polynomial and logarithmic terms or combinations thereof, and analyzing the application at a fine-grained level such as individual function calls allows the performance contributions of each input parameter to be measured even with only a few measurements. The default search space spans exponents n = 5 and J = {0, 1, 2}, meaning scalability worse than p³·log²(p) is not distinguished—a choice reflecting common behavior in parallel programs that can be tailored when expectations are known. The workflow gathers at least five data points by varying a parameter of interest while keeping everything else constant, repeats measurements depending on system noise until confidence intervals are acceptable, then generates an individual model for each kernel in the call tree by fitting coefficients for all complexity expressions spanned by the PMNF using linear regression, selecting the best model via cross-validation and ranking kernels by asymptotic complexity. Model generation costs at most seconds; the experiments dominate.

The precision of this approach has been demonstrated on real codes. Reproducing analytical models of the Sweep3D neutron transport problem, Extra-P confirmed the previously computed analytical ones, with prediction errors remaining small even at a scale of more than 100 times the training data. Comparing models trained on runs up to 2,048 processes with models trained up to 8,192 processes showed that only the coefficients of the terms, not the terms themselves, slightly change—meaning Extra-P correctly captures asymptotic behavior. In the HOMME dynamical core of the Community Atmospheric Model, developed at NCAR and confirmed to work well up to 100,000 processes, a surprising discovery emerged: the MPI_Reduce call scales significantly worse than linearly with process count. The routine sits in HOMME’s initialization, outside the code developers considered performance critical, underscoring the need to examine entire applications. Subsequent analysis revealed the bug was caused by not adapting the number of MPI processes assigned to initial I/O to the total process count.

Multi-parameter modeling posed a harder problem. With a shared search space for all parameters, the combinatorial explosion would lead to over four billion model hypotheses for the default exponents and three parameters, meaning hours per function and metric on commodity hardware. A modified golden section search—ordering hypotheses by the magnitude of their first derivative at the observation with the largest input value and traversing the space to minimize error—reduces the search from billions to less than a hundred hypotheses for a three-parameter model. Evaluation on 100,000 synthetic samples showed the heuristic differs from the exhaustive approach in less than 0.6% of cases, and even then both identify the correct lead-order term, while modeling time drops from 107 hours to 1.5 hours. An iterative refinement variant replaces fixed search spaces by starting with integer exponents and refining them through fractions with growing denominators, using the mediant of two fractions and terminating when the rate of improvement in the scale-independent SMAPE error metric sufficiently decreases.

Reducing experiment cost became a central theme. Taint analysis through the Perf-Taint tool, built on LLVM, identifies which functions and loops are affected by performance-critical parameters, allowing Extra-P to prune false dependencies and skip modeling of constant functions—in LULESH and MILC, hundreds of functions were pruned statically or dynamically. Sparse modeling exploits the assumption that each parameter exhibits a single consistent behavior across the measured space, requiring only five measurement points per parameter plus one additional point to determine whether parameter effects are independent or compounded, instead of the 5^m points of a dense matrix. A reinforcement learning agent trained on hundreds of thousands of synthetic functions yielded a cheapest-points-first strategy; on average, sparse modeling combined with this strategy reduces modeling cost by 85% while retaining 92% of model accuracy. Gaussian process regression then refined point selection further, accounting for noise levels, repetitions, and parameter ranges, improving average accuracy by up to 64.29% within the same budget by selecting fewer but more informative measurement points.

Noise, the constant of all empirical measurements, demanded its own countermeasures. Deep neural networks trained on noisy data can learn to adapt to perturbations, so an adaptive modeler estimates the noise level using a range-of-relative-deviation heuristic and selects between regression and DNN modeling accordingly. The DNN predicts PMNF exponents as a classification task over 43 classes of common performance behaviors, with coefficients determined via regression afterward, and transfer learning specializes a pretrained network to each modeling problem. Software counters offer a complementary route: counting executed LLVM basic blocks and transferred message sizes yields measurements inherently resilient to noise, which are combined with noise-sensitive wall-clock measurements through prior models of MPI operations to fit coefficients with a single noise-sensitive measurement set instead of multiple repetitions. Segmented modeling detects piecewise behaviors—such as MPI implementations switching algorithms with process count—by sliding a window across data points and comparing subset models, while an expectation-validation framework lets users specify asymptotic growth rates in big O notation and automatically generates divergence models explaining discrepancies between expected and actual performance, successfully applied to MPI libraries and parallel sorting benchmarks.

The tool’s reach now extends beyond finding scalability bugs. Hardware-independent requirement models for memory footprint, computation, communication, and memory access support co-design: modeling five scientific applications including Kripke, LULESH, MILC, Relearn, and icoFoam showed how per-process requirements change when a system is upgraded—for LULESH, doubling the racks would grow computation and communication requirements by roughly a factor of 1.2 while memory footprint stays constant, augmenting traditional back-of-the-envelope calculations for system purchasing decisions. Task-graph modeling captures isoefficiency in task-based programs like Cilk and OpenMP by comparing actual, contention-free, and upper-bound efficiency functions through a task replay engine that simulates tasks by busy-waiting for measured durations. Compositional performance modeling splits programs along natural boundaries into modules with composition operators for sequential and parallel concatenation, tracking latency and throughput—revealing, for instance, that bubblesort delivers its first sorted elements earlier than mergesort even though mergesort wins on total time. The authors’ outlook emphasizes integration with AI-driven methods and adaptation to increasingly heterogeneous hardware, where understanding how code behaves differently on CPUs versus GPUs is crucial for accurate predictions in mixed-device systems.

Subject of Research: Computer Science

Article Title: Extra-P—Empirical performance modeling made easy

Article References: Calotoiu, A., Copik, M., Czappa, F., Geiss, A., de Morais, G., Ritter, M., Shudler, S., Hoefler, T., & Wolf, F. (2026). Extra-P—Empirical performance modeling made easy. Frontiers in High Performance Computing, 3, Article 1714042. https://doi.org/10.3389/fhpcp.2025.1714042

Image Credits: AI Generated

DOI: 10.3389/fhpcp.2025.1714042

Keywords: performance modeling, high-performance computing, scalability, machine learning, software engineering, Extra-P, Empirical, performance, modeling, made, easy, scientific research