How HPC storage stacks silently lose the meaning of your data

Every time a supercomputer application saves a file, the data pass through a tower of software and hardware layers, each speaking its own dialect of input and output. A new analysis by researchers at Dresden University of Technology and the University of Göttingen, published in Frontiers in High Performance Computing, argues that this tower has a hidden flaw: as requests travel downward, the meaning behind them—their I/O semantics—evaporates. By the time a write command reaches the storage hardware, the system no longer knows whether it was part of a checkpoint that must survive a node failure, a one-off log entry, or a dataset that thousands of processes will read concurrently. The authors call this the semantic gap, and they show it is not a minor inconvenience but a structural drag on both performance and correctness in high-performance computing.

The problem begins with the architecture itself. Modern HPC I/O stacks are highly heterogeneous: an application sits at the top, followed by middleware libraries such as HDF5 or MPI-IO, then system-level interfaces like POSIX, then parallel file systems such as Lustre or GekkoFS, and finally the physical medium—NVMe solid-state drives, NVRAM, disks, or tape. Each layer translates the operations it receives into more primitive operations for the layer below, applying optimizations like aggregation, reordering, buffering, and caching along the way. But the interfaces between layers carry only syntax, not intent. A compiler can verify that a program is syntactically correct; nothing verifies that the semantic meaning of an I/O operation survives the translation. Each layer, the researchers conclude, acts like a semantic filter for the layers beneath it.

To make this concrete, the team constructed a six-category taxonomy of I/O semantics: concurrent access, persistency, consistency, spatiality, temporality, and mutability. Concurrent access describes whether a resource is touched by one process or many. Persistency captures how durable data must be—whether it lives only for the application’s lifetime, for the duration of a job, or permanently on system-level storage. Consistency defines when updates become visible to other operations, ranging from strict sequential consistency to commit-based, session-based, or eventual consistency. Spatiality describes access patterns such as contiguous, strided, or random offsets. Temporality captures reuse behavior—whether data are read once, periodically, or at random intervals—while mutability describes whether data are read-only, overwritten, or appended. Together, these categories form a common language for comparing interfaces that were never designed to be compared.

Applying this taxonomy reveals a striking asymmetry. Middleware libraries are the most expressive layer of the entire stack. MPI-IO, the I/O standard of the Message Passing Interface, is the only interface in common use that can explicitly express parallel access to a file, coordinating collective operations across a defined group of processes. It can also convey spatiality through derived data types and file views, temporality through hints such as read_once and write_once, and mutability through open-mode flags. HDF5, layered above MPI-IO, adds hyperslabs and chunking to describe non-contiguous data selections. But the richness stops abruptly at the system level. POSIX I/O—the de facto interface of HPC storage—was designed in the late 1980s for local file systems and exposes data as an unstructured stream of bytes addressed by integer file descriptors. It has no notation for shared or exclusive access, no notion of persistency requirements, and only limited hints through the posix_fadvise call.

The consequences become vivid in the checkpointing use case the authors use as their running example. Checkpointing—writing an application’s internal state to storage so it can recover from failures—is ubiquitous in scientific simulations and AI training alike. The researchers trace three plausible stacks: HDF5 over MPI-IO over POSIX over Lustre; a leaner stack of POSIX over Lustre; and POSIX over GekkoFS, a job-temporal file system built on node-local storage. In the first stack, MPI-IO can carry concurrency and spatiality information downward and even invoke Lustre-specific calls to configure striping. In the third stack, the picture is bleaker: GekkoFS provides only job-level persistency, meaning checkpoint data written to node-local storage cannot survive a node failure unless explicitly staged out to more durable storage. Had persistency semantics been transportable through the stack, the system could at least have issued a warning. As it stands, the user must simply know which file system meets which requirement.

Persistency is perhaps the most glaring casualty of the semantic gap. According to the analysis, persistency semantics are defined solely by the storage layer, yet no higher-layer interface—neither POSIX, nor MPI-IO, nor HDF5—offers a way to declare how durable data must be. The requirement is effectively lost at the application level, before the data have moved anywhere. Consistency suffers a related fate. POSIX mandates sequential consistency, which in a distributed system forces expensive synchronization and distributed locking. Yet prior studies cited by the authors found that 16 of 17 examined HPC applications could run correctly under weaker consistency guarantees than POSIX demands. File systems such as PVFS, OrangeFS, and GekkoFS already relax these guarantees—GekkoFS, for instance, offers eventual consistency for directory operations while keeping sequential consistency for file data—but because no interface can propagate consistency requirements downward, a higher layer may assume a stronger model than the lower layer provides, risking undefined behavior, or enforce needless synchronization, sacrificing performance.

At the bottom of the stack, the situation grows starker still. Modern NVMe solid-state drives process commands through multiple parallel queues and may execute them in any order; a write to position X and a read from position X issued simultaneously can complete in either sequence. The hardware, in effect, operates under eventual consistency, with no concept of files, processes, or atomicity. Any ordering guarantee must be enforced by the operating system or higher software layers, typically by flushing queues. By the time I/O reaches the device firmware, virtually all semantic context has vanished—the hardware is optimized for throughput and persistence, offering high performance predictability but the least semantic awareness of any layer in the stack.

The authors are careful to frame their contribution as analysis rather than prescription. They do not propose a new interface or implementation. Instead, they sketch a conceptual design for a semantics-aware I/O stack, in which higher layers annotate operations with explicit semantic descriptors drawn from their six categories, and these descriptors propagate through the stack so lower layers can adapt. A parallel file system might skip costly locking when an application guarantees exclusive access. A caching layer could tune prefetching and eviction based on declared reuse expectations. A job-temporal file system could trigger a warning or an automatic stage-out policy when system-level persistency is required but unsupported. The model would be declarative—intent conveyed alongside operations through metadata, configuration hints, or API extensions—rather than requiring every layer to guess what the application meant.

The study also situates itself within a broader research landscape. Earlier work by Wang and colleagues established that most HPC applications tolerate weaker consistency than POSIX provides, and the present authors previously built a tool showing that strict write consistency for data operations is often unnecessary, while parallel metadata operations such as concurrent file creations remain a real performance bottleneck. Other efforts attack the gap from different angles: Mimir interprets user intent as a proxy across the I/O stack; LabStor builds modular, user-space I/O stacks to replace rigid kernel paths; LABIOS uses a label-based architecture to serve conflicting workloads on one system. Lustre itself keeps adding features—metadata write-back caching, prefetching directory walks, persistent client caching on local SSDs—that mitigate the gap, though they require interaction through separate tools and configuration rather than through the I/O interface itself.

What emerges from the analysis is a map of exactly where meaning drains away. Middleware libraries can express the most semantics and could be extended further. MPI-IO alone can express parallel access. Persistency requirements cannot be expressed anywhere above the storage layer. Consistency models are defined piecemeal by each interface with no standardized way to declare what an application actually needs. System-level interfaces remain designed for single-node use. The authors argue that closing this gap—making semantic intent transparent and actionable across the stack—could yield I/O systems that are simultaneously faster and more correct, because storage layers would stop making pessimistic assumptions about workloads they cannot see. For a field where every wasted microsecond of I/O on an exascale machine translates into enormous cost, teaching storage stacks to understand what applications mean, not just what they ask, may prove one of the most consequential optimizations available.

Subject of Research: A taxonomy and comparative analysis of input/output semantics across the layers of high-performance computing storage stacks

Article Title: An analysis of the I/O semantic gaps of HPC storage stacks

Article References: Oeste, S., Höhn, P., Kluge, M., & Kunkel, J. (2025). An analysis of the I/O semantic gaps of HPC storage stacks. Frontiers in High Performance Computing, 3, Article 1393936. https://doi.org/10.3389/fhpcp.2025.1393936

Image Credits: AI Generated

DOI: 10.3389/fhpcp.2025.1393936

Keywords: high-performance computing, I/O semantics, storage stacks, POSIX, MPI-IO, HDF5, parallel file systems, Lustre, GekkoFS, consistency models, checkpointing, semantic gap