Ad-Hoc Parallel File System Boosts Big Data Analytics Speed Up to 4.5-Fold

A team of researchers at Universidad Carlos III de Madrid has demonstrated that a purpose-built, ad-hoc parallel file system can dramatically accelerate big data analytics workloads, outperforming two of the most widely deployed storage platforms in the field. In a study published in the Journal of Big Data, the group, led by Gabriel Sotodosos-Morales and including Felix Garcia-Carballeira, Diego Camarmas-Alonso, Alejandro Calderon-Mateos, Dario Muñoz-Muñoz and Jesus Carretero, paired the Apache Spark analytics engine with Expand, a parallel and distributed file system developed by the university’s ARCOS research group. Their benchmarks showed that Expand delivered performance up to 4.5 times higher than Lustre and up to 2.5 times higher than the Hadoop Distributed File System, better known as HDFS, on the widely used TeraSort benchmark. The finding challenges a long-standing assumption that general-purpose parallel file systems are the best foundation for data-intensive computing, and it points toward a new generation of storage software designed specifically for the access patterns of modern analytics.

The motivation behind the work stems from a fundamental shift in the demands placed on high-performance computing infrastructure. Data processing in areas such as artificial intelligence and big data has, in recent years, pushed traditional HPC systems well beyond their original design envelopes. Analytics frameworks like Spark process enormous volumes of data through massive input and output operations, and these I/O operations frequently become the dominant bottleneck in the entire pipeline. While researchers have made remarkable progress in speeding up computation through in-memory processing and distributed execution, the storage layer beneath these frameworks has often lagged behind. When thousands of tasks simultaneously read and write terabytes of intermediate results, the file system can become the slowest component in the cluster, leaving expensive processors idle while they wait for data to arrive.

Parallel file systems have long been the standard answer to this challenge in HPC environments. Systems such as Lustre stripe files across many storage servers so that multiple clients can read and write data simultaneously, and techniques such as file partitioning and replication provide both high performance and high availability. HDFS, meanwhile, was designed specifically for the Hadoop ecosystem, replicating blocks across commodity nodes to achieve fault tolerance and data locality. Yet both systems were conceived for workloads that differ in important ways from those generated by modern analytics engines. Spark, in particular, was designed around in-memory processing and resilient distributed datasets, and its I/O behavior, characterized by many small, concurrent, short-lived read and write operations, does not always align with the assumptions built into these general-purpose storage platforms.

This is where the concept of an ad-hoc parallel file system comes in. Rather than serving as a permanent, cluster-wide storage fabric, an ad-hoc file system is deployed on demand alongside a specific analytics job, using the same nodes that run the computation. Expand, the file system at the heart of the new study, is a parallel and distributed file system designed by the ARCOS research group that can operate in precisely this mode. By co-locating storage with computation, Expand eliminates much of the network traffic and metadata overhead that conventional file systems incur when data must be fetched from dedicated storage servers. The researchers describe this approach as a way to alleviate the I/O bottlenecks that arise in traditional parallel file systems when they are asked to serve big data analytics workloads.

The technical integration with Apache Spark is a central contribution of the work. Spark is written in the Java programming language and runs on the Java virtual machine, which means that a file system written in a language such as C or C++ cannot be called directly by the framework. The researchers addressed this by exposing Expand to Spark as a Hadoop-compatible file system, allowing the analytics engine to interact with Expand through the standard interfaces it already uses, with a Java native interface bridging the gap between the two software layers where needed. This design means that existing Spark applications can take advantage of Expand without being rewritten, preserving the benefits of both platforms: Spark’s powerful distributed computation model and Expand’s high-performance storage architecture.

Under the hood, Expand relies on the same principles that make modern parallel file systems effective, including file partitioning and replication, but it applies them with the specific access patterns of analytics workloads in mind. Data is distributed across the storage servers that participate in the ad-hoc deployment, so that read and write operations can proceed in parallel across many disks and network paths simultaneously. Replication ensures that the failure of a single node does not compromise the availability of the data during a job’s execution. The combination is intended to provide highly available, high-performance storage precisely when and where an analytics application needs it, and to be dismantled just as quickly when the job is done.

To evaluate the approach, the researchers compared Expand against two heavyweight competitors: Lustre, one of the most widely used parallel file systems in supercomputing centers around the world, and HDFS, the storage backbone of the Hadoop ecosystem. The evaluation used TeraSort, a benchmark that has become a de facto standard for measuring the end-to-end performance of big data platforms because it exercises sorting, shuffling, and both reading and writing of very large datasets. The results were striking. On this benchmark, Expand achieved performance up to 4.5 times higher than Lustre and up to 2.5 times higher than HDFS. These are not marginal improvements; in a field where a few percentage points of speedup can translate into significant cost savings, a several-fold improvement suggests that the storage layer, rather than the compute layer, was the dominant constraint in the tested configurations.

The implications of the study extend beyond a single benchmark. As artificial intelligence and big data applications continue to grow in scale and importance, the cost of I/O bottlenecks compounds across every stage of the data lifecycle, from ingestion and cleaning to model training and result aggregation. The finding that an ad-hoc file system, deployed only for the duration of a job and co-located with the computing resources, can substantially outperform permanently provisioned storage infrastructures suggests that analytics clusters could be re-architected around tighter integration of storage and computation. It also highlights a potential inefficiency in the common practice of running analytics frameworks on top of storage systems designed for other purposes, whether for traditional HPC simulation workloads in the case of Lustre or for batch processing in the case of HDFS.

The work forms part of a broader research effort at the ARCOS group to develop high-performance storage for hybrid HPC and data-intensive workloads. The research was partially supported by the Spanish Ministry of Science and Innovation through the project Expand: High Performance Storage System for HPC and Big Data Environments, funded under the European Union NextGenerationEU and PRTR programs, and through the SCIOT project on new scalable I/O techniques for hybrid workloads. The authors also acknowledged the use of the C3 computer resources provided by the Centro para el Análisis y Modelado de Sistemas Complejos en Ingeniería y Biomedicina. The study is open access, and the researchers report no competing interests.

For practitioners, the message of the research is that the storage layer of a big data analytics ecosystem is not a fixed constraint but a design choice with real performance consequences. As data volumes continue to explode across fields from genomics to climate modeling to commercial machine learning, solutions that treat storage as an integral, workload-aware component of the analytics pipeline, rather than a passive repository, may define the next generation of data platforms. The Madrid team’s results with Expand provide a concrete, benchmarked demonstration that rethinking the file system itself, and adapting it to the specific rhythms of frameworks like Apache Spark, can unlock performance that conventional architectures leave on the table.

Subject of Research: Ad-hoc parallel file systems for accelerating big data analytics workloads in Apache Spark environments.

Article Title: Improving big data analytics ecosystems using ad-hoc parallel file systems

Article References: Sotodosos-Morales, G., Garcia-Carballeira, F., Camarmas-Alonso, D., Calderon-Mateos, A., Muñoz-Muñoz, D., & Carretero, J. (2026). Improving big data analytics ecosystems using ad-hoc parallel file systems. Journal of Big Data. https://doi.org/10.1186/s40537-026-01559-6

Image Credits: AI Generated

DOI: 10.1186/s40537-026-01559-6

Keywords: big data analytics, parallel file system, Apache Spark, Expand, HDFS, Lustre, high-performance computing, I/O bottlenecks, TeraSort, distributed storage, in-memory processing, ad-hoc file system