Score-P was born in 2009 out of frustration. Research teams from six organizations that each maintained their own performance analysis tools realized they were duplicating enormous amounts of work: every new HPC system and programming interface forced each team to separately update their proprietary measurement infrastructures, leaving little time to improve the actual analysis features. Worse, the tools used incompatible output formats, so application developers had to learn and repeatedly apply several measurement systems with subtle differences in usage just to benefit from complementary analyses—a pain point that surfaced repeatedly during joint training sessions of the Helmholtz Virtual Institute for High Productivity Supercomputing. The teams joined forces, with initial funding from the German Federal Ministry of Education and Research and the U.S. Department of Energy, to build a single, unified measurement layer. Since then, Score-P has been widely adopted at HPC sites worldwide, serves as a key data acquisition tool for the European Performance Optimisation and Productivity Centre of Excellence, and has been taught to more than 1,000 users at tuning workshops, tutorials at major conferences such as SC and ISC, and international summer schools.

At its heart, Score-P solves a deceptively simple problem: how do you see what a massively parallel program is actually doing? The answer is instrumentation—injecting measurement hooks into an application at relevant points—and measurement, capturing what happens when those hooks fire. Score-P supports two complementary modes. Profiling aggregates statistics over the program’s execution at roughly constant memory cost, while event tracing chronologically records individual execution events, consuming memory proportional to the run’s duration but enabling far deeper analysis of dynamic behavior. For C, C++, and Fortran codes, instrumentation is remarkably lightweight: prefixing the compiler with the scorep command handles both compilation and linking. The instrumenter automatically intercepts function entry and exit points, applies heuristics to detect which parallel programming interfaces are in use, and selects the appropriate measurement libraries at link time. Users can also manually annotate code regions, or even measure unmodified binaries through dynamic interposition and sampling.

The architecture that has emerged over more than a decade is deliberately modular. Early releases supported only four application types through four monolithic measurement libraries—an approach that proved too restrictive as new APIs proliferated. The developers refactored the system so that event sources, data sources, and measurement core components are freestanding and flexibly composable, communicating through well-defined interfaces. When an instrumentation hook fires, an adapter maps the activity to an abstract event model and passes it to the measurement core, which augments the event with high-accuracy timestamps and, optionally, hardware counter data from sources such as PAPI, getrusage, or Linux perf. The enriched events then flow to event consumers, called substrates. Two built-in substrates produce the ecosystem’s two standard output formats: CUBE4 for profiles and OTF2 for traces. Both plug-in mechanisms—custom substrates and custom metric sources—allow the community to extend the system without touching the core.

The data model itself is worth understanding, because it is what makes cross-tool interoperability possible. Definitions capture static entities—source code regions, processes, threads, MPI communicators—while events capture what happens on a given location at a specific time. For multi-process experiments, process-local definitions are merged into a global set through a tree-based unification algorithm that runs in parallel with logarithmic complexity after measurement completes. The system tree separately models the underlying hardware, from cores and sockets to network components, and Cartesian topologies make large-scale system trees more intuitive to navigate. Events are typically paired—Enter and Leave on a single location, Send and Receive across locations—giving analysis tools a consistent semantic foundation regardless of which adapter produced the data.

That foundation supports a remarkably rich ecosystem. Cube provides interactive profile analysis through coupled tree browsers spanning metrics, call paths, and system resources, with color-coded hotspots, violin plots, and a client-server mode for reports too large for a workstation; its own scripting language, CubePL, lets users derive new metrics from raw data. Extra-P attacks scalability from a different angle: from a series of measurements with varied parameters—five distinct values at minimum—it builds empirical performance models in a restricted normal form of polynomial and logarithmic terms, fitted by linear regression, to predict how run time will change as the machine grows and to flag scalability bottlenecks before they burn precious computing resources. TAU, the veteran Performance System from Oregon, can operate standalone or integrate with Score-P on both the measurement and analysis sides, and offers a distinctive ability to track idle time spent in implicit barriers within MPI collective operations, exposing synchronization problems that ordinary profiling misses.

Scalasca and Vampir round out the core ecosystem. Scalasca’s automatic trace analyzer runs as a parallel program that replays the captured trace, identifying wait states in communication and synchronization, pinpointing the imbalances that cause them, and computing the critical path of execution—results delivered as enhanced CUBE4 profiles that highlight optimization candidates. Once Score-P existed, Scalasca’s own measurement system was retired in favor of the common OTF2 and CUBE4 formats. Vampir, a commercial visualization tool, consumes OTF2 traces directly and centers on a master timeline plotting events across ranks and time, with synchronized summary charts, communication matrices, and call trees at every zoom level; it can run in client-server mode where the server executes as an MPI application on the HPC system itself. Vampir has recently extended to scientific workflow executions using the community-driven WfFormat, visualizing task dependencies derived from input and output files.

Keeping this machinery current with a rapidly changing hardware landscape has demanded constant reinvention. The MPI adapter now covers all of MPI 4.0—roughly 1,200 wrapper functions across C, Fortran 90, and Fortran 2008 bindings, generated automatically from the MPI Forum’s machine-readable specification—with recent work adding internally consistent measurement of MPI_THREAD_MULTIPLE programs and ongoing efforts toward matched messages and partitioned communication. For OpenMP, the source-to-source instrumenter OPARI2, which predates any official tools interface, is now joined by OMPT, the callback-based tools interface introduced with OpenMP 5.0 and the default in Score-P since version 9.0; detecting runtimes that deviate from the specification proved so important that the team built a dedicated test tool, ompt-printf, and filed roughly 80 bug reports with compiler vendors between December 2022 and May 2025. On the accelerator front, Score-P 9.0 covers CUDA, HIP, OpenMP Target, OpenACC, OpenCL, and Kokkos, modeling GPU streams as locations and correlating host-side launches with device-side events through hash-based identifiers, while carefully handling the out-of-order event delivery that buffer-based tools interfaces can produce.

Perhaps the most technically elegant recent addition is the hybrid recording mode combining instrumentation with sampling. Because automatic compiler instrumentation can impose significant overhead, Score-P introduced stack unwinding and sampling: interrupts triggered by a counter—time-based by default, but any PAPI or perf counter works—capture call stacks outside instrumented API calls, and new CallingContextEnter, CallingContextLeave, and CallingContextSample events weave these samples into a calling context tree. Compiler-instrumented events are suppressed in this mode, meaning a single Score-P-instrumented binary can serve both instrumentation-based and sampling-based measurements without recompilation. Compiler plug-ins for GCC and, since version 9.0, LLVM enable compile-time filtering that excludes inlined functions and tool internals, and a prototype adapter based on LLVM’s XRay function tracing system promises near-zero-overhead run-time filtering as a middle ground.

None of this would be reliable without an unusually rigorous engineering culture. The distributed team invested early in continuous integration and continuous deployment, building and testing every merge request in container environments and then deploying tarballs to real HPC systems spanning x86-64 and ARM CPUs with NVIDIA, AMD, and Intel accelerators. Run-time testing is deliberately outsourced to feature-specific suites, including a modified version of the MPICH test suite with over 1,000 tests and the OpenMP specification’s example collection. Looking forward, the authors see heterogeneity itself as the defining challenge: tools interfaces remain an afterthought for many APIs, task-based parallelism still defeats flat-list profile presentation in all existing tools, and the team is proposing an RDF-based metadata extension to bring FAIR principles—findability, accessibility, interoperability, and reusability—to performance data. With more than 1.8 million concurrent threads already demonstrated, Score-P’s bet is that community-driven, vendor-neutral measurement infrastructure will remain essential as exascale systems push every layer of the software stack to its limits.

Subject of Research: The Score-P instrumentation and measurement infrastructure and its ecosystem of HPC performance analysis tools

Article Title: The Score-P performance tools ecosystem

Article References: Feld, C., Calotoiu, A., Corbin, G., Geimer, M., Hermanns, M.-A., Knespel, M., Mohr, B., Reuter, J. A., Sander, M., Saviankou, P., Schlütter, M., Schöne, R., Shende, S. S., Visser, A., Wesarg, B., Williams, W. R., Wolf, F., Wylie, B. J. N., & Zarubin, M. (2026). The Score-P performance tools ecosystem. Frontiers in High Performance Computing, 3, Article 1709051. https://doi.org/10.3389/fhpcp.2025.1709051

Image Credits: AI Generated

DOI: 10.3389/fhpcp.2025.1709051

Keywords: Score-P, high-performance computing, exascale computing, performance analysis, profiling, event tracing, MPI, OpenMP, GPU accelerators, Scalasca, Vampir, performance modeling