The research, published in Frontiers in High Performance Computing, draws an unusual analogy between the world of science and the world of business. Large enterprises long ago abandoned the idea that every application should talk directly to every other application through shared databases. Instead, they adopted event-driven architectures, in which a distributed streaming service such as Apache Kafka acts as a single source of truth. Applications publish immutable events to topics, other applications subscribe to those topics, and the streaming service decouples producers from consumers in both time and space. A faulty application simply restarts and picks up where it left off in the stream. The authors describe such event stores as a database turned inside out, and they pose a provocative question: what would a parallel file system turned inside out look like?

The motivation is rooted in the changing character of HPC workloads. Scientific workflows increasingly integrate multimodal data streams from computations, simulations, and instruments, from small telemetry events to gigabyte-scale scientific datasets. Provenance capture, telemetry, and anomaly detection generate background I/O traffic from many sources. Coupling supercomputers with light sources, telescope arrays, and gravitational wave detectors introduces data-management challenges that neither parallel file systems nor tightly coupled in situ processing handle well. Parallel file systems suffer from a POSIX consistency model that forces locking and synchronization, serialization overheads, and the infamous computation-and-I/O gap that bottlenecks performance at scale. In situ approaches bypass the file system for speed but sacrifice persistence, fault tolerance, and impedance matching between components that produce data at different rates.

Mofka, built on the Mochi suite of HPC data-service components, aims to occupy the middle ground. Its design goals read like a checklist of supercomputer peculiarities: efficient use of high-performance networks such as InfiniBand, Omni-Path, and HPE Slingshot, including native RDMA support that moves data directly from application memory to storage with as few copies as possible; efficient threading on massively multicore nodes, mixing hardware threads with user-level co-routines; handling of event payloads ranging from kilobytes to multiple gigabytes; support for scientific data formats that lets consumers specify interest in only sections of multidimensional arrays before transfer; and high modularity so the network stack, scheduling policies, and storage backends can be fine-tuned per use case. Polaris at Argonne features two network interface cards per node, and Aurora features eight, and Mofka’s Plumber component automatically assigns server processes to their closest NIC to maximize network resources.

The architecture’s most distinctive decision is the division of every event into two parts. A small, structured metadata part, interpretable by both clients and servers, travels through an optimized control path: it is validated by a user-provided validator, routed by a partition selector, and serialized into batches sent via RDMA to a log provider. A potentially much larger data part, opaque to the server, travels by RDMA directly to and from user-provided memory and its final storage destination, bypassing intermediate buffers entirely. On the consumer side, a data selector decides which portion of the data to pull, all, none, or an arbitrary subset, before any transfer occurs, and a data broker tells the client library where to place it. In a weather application, one consumer might select a two-dimensional slice at ground level for a color map while another pulls the full three-dimensional chunk for volume rendering, from the same topic.

Benchmarks on Argonne’s Improv and Polaris supercomputers and Oak Ridge’s Frontier exascale machine compared Mofka against Kafka and Redpanda, which speak the same protocol. The results were revealing, and sometimes humbling. In single producer-consumer tests, Mofka outperformed both rivals in almost all scenarios on Improv, while on Polaris Redpanda performed better in production of events of one kilobyte and up. When one server served increasing numbers of client nodes, Mofka throughput doubled as server nodes doubled on Improv, while Kafka and Redpanda stagnated, apparently capped by their reliance on TCP at roughly one gigabyte per second. When events carried one kilobyte of metadata plus ten megabytes of data, using the separated data path with in-memory storage pushed production to 6,257 megabytes per second and consumption to 8,529 megabytes per second, roughly six times the metadata-only path. The team also learned that swapping glibc’s allocator for jemalloc more than doubled consumer throughput in some configurations, and that Kafka and Redpanda frequently timed out and failed to complete benchmarks as concurrency increased, making them difficult to use on HPC systems without an experienced administrator.

Three real-world applications demonstrated the framework’s utility. TekApp, a streaming tomographic reconstruction mini-app derived from the Trace code, reconstructs three-dimensional volumes from X-ray projections at the Advanced Photon Source. Its original ZeroMQ-based configuration required components to run simultaneously with frequent synchronization; with Mofka, the workflow became loosely coupled and fault tolerant, with per-event overhead reduced by at least twelve times across components and up to five hundred times better performance for large twenty-five-megabyte events between the reconstruction and denoising stages. MOFA, an AI-driven discovery pipeline for metal-organic frameworks for carbon capture, links a diffusion-based generative model with a ladder of physics simulations in a closed learning loop; replacing its single Redis instance with distributed Mofka topics eliminated a single point of failure, although experiments showed that Mofka currently suffers latency in latency-bound, small-message scenarios when Python’s threading limitations force blocking calls. Flowcept, a provenance and performance instrumentation tool, streams profiling data from a multinode, multi-GPU large language model hyperparameter search on Polaris; Mofka’s overhead ranged from 0.22 to 1.33 percent of workflow runtime, against 6.31 to 26.09 percent for Kafka and up to 36.96 percent for Redis, with Mofka spending just 7.30 seconds blocking on communication during a heavyweight profiling run while Kafka blocked for 36.19 seconds and Redis for 265.81 seconds.

The authors are candid about the lessons learned. Batch size matters enormously, and larger is not always better: for the SIRT component producing twenty-five-megabyte events, increasing batch size actually increased overhead because batching delays transfers whose performance is driven by the data path rather than the number of RPCs. More partitions equal more parallelism, even from a single process to a single server. Python bindings carry significant overhead, with object conversions and the Global Interpreter Lock often costing more than event serialization or transfer. Threading strategies, process placement, and oversubscription frequently mattered more than which streaming system was used at all. In one experiment on Polaris, oversubscribing thirty-two cores with sixteen processes and background threads dramatically changed results, and the use of an Argobots user-level thread combined with librdkafka’s POSIX threads caused interference that made Mofka’s API detrimental to consumption performance.

The comparison the authors ultimately draw is with file systems rather than messaging systems: comparing Mofka with Kafka, they write, is like comparing Lustre with NFS. The former is HPC-ready, uses high-performance networks, and offers the right features for scientific I/O; the latter is useful but will always rely on Ethernet and carry performance limitations. Kafka’s development started in 2011 and Redpanda’s in 2019, each backed by a company and hardened by enterprise use, yet a prototype built with limited resources in a short time outperforms them in many HPC scenarios, which the team sees as an encouraging first step justifying further investment. Future work will add data replication using an RDMA-enabled implementation of the RAFT protocol, authentication and access control to turn Mofka into a multiuser service, and bridges to technologies such as Octopus to form a scientific event fabric spanning instruments, supercomputers, edge devices, and the cloud. If they succeed, the humble event stream, the backbone of internet commerce, may become the backbone of computational science as well.

Subject of Research: Toward a persistent event-streaming system for high-performance computing applications

Article Title: Toward a persistent event-streaming system for high-performance computing applications

Article References: Dorier, M., Gueroudji, A., Hayot-Sasson, V., Nguyen, H. D., Ockerman, S., Souza, R., Bicer, T., Pan, H., Carns, P., Chard, K., Chard, R., Gonthier, M., Huerta, E., Lenard, B., Nicolae, B., Patel, P., Wozniak, J., Foster, I., Rao, N. S., & Ross, R. B. (2025). Toward a persistent event-streaming system for high-performance computing applications. Frontiers in High Performance Computing, 3, Article 1638203. https://doi.org/10.3389/fhpcp.2025.1638203

Image Credits: AI Generated

DOI: 10.3389/fhpcp.2025.1638203

Keywords: Mofka, event streaming, high-performance computing, RDMA, Kafka, Redpanda, Mochi, supercomputers, scientific workflows, data management, exascale, provenance