MOOSE Simulation Framework Gains GPU Acceleration Through Kokkos Port

The new capability, called Kokkos-MOOSE, is described in a paper published in SoftwareX and arrives as an optional extension rather than a rewrite, meaning existing MOOSE applications can adopt GPU acceleration largely by swapping object names in their input files.

MOOSE is a finite element framework for building domain applications, most prominently in nuclear engineering, and it relies on libMesh for finite element machinery and PETSc for scalable nonlinear and linear solvers. Until now, MOOSE has scaled to large problems using Message Passing Interface parallelism on multi-core CPUs. That approach is increasingly out of step with modern high-performance computing systems, where most of the performance comes from GPUs rather than processors.

The design challenge was substantial. MOOSE’s flexibility rests on dynamic polymorphism, a mechanism in which base classes call virtual functions that derived classes override, letting developers plug in new physics without touching framework code. Virtual function tables carry restrictions and performance costs on GPU backends, yet simply discarding the template method pattern would break the extension points that the ecosystem's existing applications, 53 tracked applications as of June 2026, depend on. Kokkos-MOOSE solves this with static polymorphism: the hooks that derived classes implement become public template methods marked with the KOKKOS_FUNCTION annotation, and thread-private indices and context are passed as arguments rather than stored as members. Because the call is made on the concrete object, ordinary C++ name lookup binds directly to the derived class method, mimicking overriding without a virtual function table.

To connect statically typed device code with the runtime object model, the framework registers dispatchers for each object. A dispatcher is a template class parameterized by the object type and an operation tag that stores both a host reference to the object and a device-accessible copy of it. It acts as a Kokkos functor whose parallelFor launches the loop, while deriving from a type-erased base class with a virtual parallelFor so host code can launch it without knowing the concrete type at compile time. The framework’s factory, warehouse, action, executioner, and input/output systems still construct and coordinate simulations on the host, preserving the familiar workflow in which users define physics by deriving kernels, boundary conditions, and materials and compose them through an input file.

The implementation builds on Kokkos, a C++ programming model for performance-portable code across diverse hardware, obtained through PETSc. Source files carrying the dedicated .K extension are compiled with the GPU toolchain while other MOOSE sources use the standard CPU toolchain, so downstream applications never need to be rebuilt with GPU compilers. Preprocessor guards expose Kokkos-specific functionality conditionally, and custom data containers such as Moose::Kokkos::Array maintain consistent class layouts across the separate compilation boundary. A value-binding discipline replaces direct references for device-side data, with lightweight handles and wrappers that refresh cached device copies through their copy constructors, exploiting the fact that Kokkos copies the functor before each parallel dispatch.

The authors demonstrated the capability on a transient heat conduction simulation of a 5 megawatt-thermal microreactor, modeled with one-twelfth symmetry using 190,744 elements and a second-order Lagrange temperature basis yielding 1,394,008 degrees of freedom. Heat pipes were represented by convective boundary conditions at a bulk temperature of 800 K, exterior surfaces received convective and radiative conditions at 300 K, and gap regions were simplified as low-conductivity solids because gap heat transfer kernels have not yet been ported. The transient ran for 40,000 seconds in 61 time steps and reached steady state under a uniform power density distribution.

On this problem, run with the Newton-Krylov method and solved iteratively through PETSc, the GPU calculation used a single NVIDIA H200 while the CPU calculation used two Intel Xeon Platinum 8480+ processors totaling 112 cores. Residual evaluation ran 10.0 times faster and Jacobian assembly 27.4 times faster on the GPU, with total computing time falling from 392.6 seconds to 66.3 seconds, a 5.9-fold speedup equivalent to 663 CPU cores under perfect scaling. Nonlinear iteration counts were nearly identical between the two runs, but the GPU required roughly 2.5 times more linear iterations, which the authors attribute to differences between the CPU and GPU implementations of the HYPRE BoomerAMG preconditioner. Future work will focus on tuning PETSc and HYPRE parameters for GPUs.

The paper is candid about trade-offs. When Kokkos-MOOSE runs on CPUs against the original MOOSE on the same 112 cores, it is about 20 percent slower overall, mainly because Jacobian evaluation takes 50 percent longer, and it uses slightly more memory, 132.9 gigabytes versus 126.9. The reason lies in execution patterns: the original MOOSE computes every object in a single monolithic element loop, caching shape functions and material properties for one element at a time, whereas each Kokkos-MOOSE object runs its own parallel loop for device-side static polymorphism and must cache such data globally. The authors therefore position the CPU backend primarily as a testing and debugging tool that emulates GPU execution rather than a full replacement.

A second demonstration coupled steady-state neutronics with heat conduction through MOOSE’s MultiApp and Transfer systems. The neutronics side was handled by Griffin, a MOOSE-based reactor physics application itself ported to GPUs using Kokkos, running a discontinuous finite element discrete ordinates transport solver with 11 energy groups, 144 angular directions, and P3 scattering, amounting to about 1.25 billion angular flux degrees of freedom. The two applications exchanged power density and temperature through Picard iteration. Neutron transport achieved a 6.3-fold speedup and dominated the runtime, giving an overall 5.4-fold speedup equivalent to 604 CPU cores. The heat conduction sub-application was too small to benefit because one-time setup costs dominated, and transfers still run on CPUs and were slower in the GPU calculation.

Kokkos-MOOSE occupies a distinct position among performance-portable finite element efforts such as MFEM, Albany, libCEED, and deal.II. In MFEM and Albany, developers extending the systems must implement both the local physics and the portable parallel loop applying it, while libCEED relies on just-in-time compilation of user-provided point-wise functions and deal.II requires device-callable functors composed through C++ templates. Kokkos-MOOSE instead places element traversal, quadrature loops, data preparation, and assembly in reusable framework base classes, so derived physics objects implement only local weak form evaluations, closely following the programming model of their original MOOSE counterparts, and independently developed objects compose at runtime through MOOSE’s pluggable object model.

The current version supports a broad set of base classes, including kernels, boundary conditions, nodal kernels and kernels for linear finite volume systems, auxiliary kernels, materials, user objects, postprocessors, vector postprocessors, reporters, and functions, with several families supporting automatic differentiation through a Kokkos port of MetaPhysicL. Not everything has been ported: mesh displacement and adaptivity, discontinuous Galerkin kernels, and mortar finite elements are not yet supported and will be added progressively. Multi-GPU execution is supported through MOOSE’s existing distributed mesh and equation system parallelization, and restart, execution ordering control, and coupling with other applications all remain available.

The stakes extend beyond one laboratory codebase. As of June 2026, the MOOSE ecosystem comprises 25 community-developed physics modules and 53 tracked applications, including more than a dozen production-grade applications vital to nuclear energy research, representing what the authors describe as hundreds of person-years of investment in infrastructure, verification, and validation. Reinventing those capabilities outside MOOSE purely for GPU acceleration would risk fragmenting decades of accumulated expertise. By letting developers incrementally port individual objects while retaining the framework’s input/output and testing systems, mesh generation, assembly, and solver integration, Kokkos-MOOSE offers the ecosystem a path to evolve alongside emerging HPC architectures without sacrificing its mature software infrastructure.

Subject of Research: GPU acceleration of the MOOSE multiphysics finite element framework using the Kokkos performance-portability model

Article Title: Kokkos-MOOSE: GPU-based multiphysics simulation in the MOOSE framework

Article References: Choi, N., Lindsay, A., Harbour, L., Chowdhury, R., Stogner, R., & Permann, C. (2026). Kokkos-MOOSE: GPU-based multiphysics simulation in the MOOSE framework. SoftwareX, 36, Article 103156. https://doi.org/10.1016/j.softx.2026.103156

Image Credits: AI Generated

DOI: 10.1016/j.softx.2026.103156

Keywords: MOOSE, Kokkos, GPU computing, multiphysics simulation, finite element method, high-performance computing, static polymorphism, nuclear engineering, PETSc, performance portability, reactor simulation, scientific software