Terminological decision trees sit at the intersection of two venerable research traditions. On one side is the decision tree, the workhorse of classical machine learning, in which a cascade of yes-or-no tests sorts data into categories and can be read off directly by a human. On the other side is Description Logic, the family of formal languages that underlies the Semantic Web and the Web Ontology Language, known as OWL. In a standard decision tree, each node tests a simple feature. In a terminological decision tree, each node instead tests a logical concept, for example whether an individual belongs to the class of students who take at least one course. The tree thereby classifies individuals in a knowledge base and, in doing so, can propose new concept definitions that were never written down in the original ontology. That makes TDTs valuable not only for classification but for knowledge discovery, filling gaps in ontologies that encode domains from medicine to finance.

The catch has always been scale. Existing systems for inducing terminological decision trees, most notably the TermiTIS system and the ontology-driven decision tree approaches that followed it, run on a single machine and in memory. They were validated on ontologies with at most around 15,000 individuals. Meanwhile, the pressure to scale semantic machine learning has pushed much of the field toward sub-symbolic methods: graph neural networks, large language models and geometric ontology embeddings. These techniques distribute beautifully across clusters, but they sacrifice the strict logical guarantees that many applications cannot live without. Medical decision support, legal reasoning and regulatory compliance all demand classifiers whose reasoning can be audited step by step. Mohamed and her colleagues argue that DistTDT closes this gap, offering horizontal scalability without giving up symbolic exactness.

The architecture rests on Apache Spark, the distributed computing framework whose core abstraction, the Resilient Distributed Dataset, spreads partitioned data across worker nodes with fault tolerance. The workflow unfolds in three stages. First, the OWL layer of the SANSA framework converts the input ontology into distributed data structures, and a scalable statistics module extracts the classes, properties and individuals, partitioning them across the cluster. Second, a query generation phase produces artificial learning problems by combining two to eight primitive concepts with logical operators such as conjunction, union, negation and existential or universal restrictions. Third comes the actual tree induction, in which the driver coordinates worker nodes that evaluate candidate concepts in parallel against their local slices of instance data.

At the heart of the method lies a refinement operator, a device borrowed from inductive logic programming. Rather than randomly generating candidate concepts, as earlier approaches did, the operator systematically specializes or generalizes a parent concept to produce candidates that either subsume or are subsumed by it. Given the concept of a person who takes some course, the operator might produce a student who takes a semantic web course, or a graduate student who takes a course in both software engineering and artificial intelligence. Crucially, the operator exploits the subsumption hierarchy already encoded in the ontology’s terminology, which ensures that only refinements consistent with the existing knowledge base are generated. The number of possible specializations is in principle unlimited, so the system bounds the search breadth with a configurable parameter, preventing the combinatorial explosion from drowning the cluster in communication overhead.

Selecting which candidate concept to install as a new tree node relies on an information gain criterion computed with the Gini index, the same purity measure familiar from classical decision tree algorithms such as C4.5. Because some individuals may have uncertain membership in a concept, the authors extend the gain calculation to a ternary setting, treating positive, negative and uncertain individuals as separate impurity components. Once the best concept is chosen, the individuals are split into left and right subtrees, with uncertain individuals routed into both branches, and the algorithm recurses. A pruning threshold of 0.05 on the information gain keeps the trees from growing branches that add little statistical value, a choice the authors found balances accuracy against depth and cluster runtime.

Perhaps the most technically ambitious component is the prototype distributed reasoner that supports the whole enterprise. Inducing a terminological decision tree requires answering logical questions at every step: does this class have any instances, which individuals belong to it, and is a given assertion entailed by the knowledge base? Traditional tableau-based reasoners such as HermiT or the highly optimized Konclude are powerful but confined to a single machine, and their search spaces can grow exponentially. The new framework adopts a hybrid design. Heavy tableau mechanics, including backtracking and blocking to prevent infinite loops, run locally within each node’s memory. Above that, a distributed layer partitions logical dependencies, routes sub-queries across the cluster in parallel and aggregates the results. Candidate concepts travel to workers as broadcast variables, so the only network traffic is the stream of match counts flowing back to the driver. The authors validated the reasoner’s output with unit tests cross-checked by human domain experts.

The empirical evaluation, conducted on a five-node cluster of AMD Opteron machines with 64 cores and roughly 250 gigabytes of memory each, tested the framework on four expressive ontologies: Michalski’s classic trains problem, the medical MDM0.73 ontology, a financial ontology for e-banking and the mutagenesis benchmark. Against TermiTIS, the state-of-the-art centralized baseline, the distributed system dominated on every metric examined, including match rate, commission error rate, omission error rate and induction rate. The runtime comparison was stark. On the financial ontology, TermiTIS needed more than 340 minutes to complete its task, while DistTDT finished in under 40, roughly 11.8 percent of the baseline’s time. Consistency improved just as dramatically: on the medical dataset, the distributed system showed a standard deviation of 0.6 seconds against 72.8 for the centralized approach, with correspondingly narrower confidence intervals. The authors attribute much of this gain to the refinement operator, which replaces arbitrary concept generation with a structured traversal of the concept space.

Notably, the system sometimes classified individuals that a deductive reasoner could not decide at all, a phenomenon the authors measure as an induction rate. These induced classifications are logically new claims, not derivable from the ontology, but they may be exactly the kind of knowledge that helps populate incomplete knowledge bases, provided an ontology engineer reviews them. The team is candid about the trade-offs. Data skew from dense semantic hubs can stall tasks, though it does not affect correctness; Spark’s lineage-based fault tolerance can impose re-computation costs on node failures, mitigated here by selective caching; and distributing description logic syntax across Java virtual machines carries serialization overhead managed with the Kryo library. The approach also imposes some overhead on small, in-memory datasets where a single machine would suffice.

The researchers, whose implementation is open source and integrated into the SANSA framework as an extension built on Apache Spark, outline three directions for future work: extending the machinery beyond the current logic to more expressive OWL 2 profiles such as SROIQ, stress-testing against industrial-scale benchmarks exceeding millions of axioms, and building a dedicated benchmark for cluster scalability. For a field increasingly dominated by opaque statistical learners, the message of this work is a provocative one. Exact, auditable logical reasoning and planetary-scale data processing no longer have to be opposing choices, and the explainable branch of artificial intelligence may be ready for its own scaling moment.

Subject of Research: Distributed learning of terminological decision trees over large-scale OWL ontologies using Apache Spark and a distributed description logic reasoner.

Article Title: DistTDT: distributed terminological decision tree learning

Article References: Mohamed, H., Kniesel-Wünsche, G., Lehmann, J., & Fathalla, S. (2026). DistTDT: distributed terminological decision tree learning. Knowledge and Information Systems, 68(1), Article 263. https://doi.org/10.1007/s10115-026-02891-2

Image Credits: AI Generated

DOI: 10.1007/s10115-026-02891-2

Keywords: DistTDT, terminological decision trees, description logic, OWL, Apache Spark, distributed computing, concept learning, refinement operator, distributed reasoning, semantic web, explainable AI, SANSA framework

The motivation behind the work stems from a fundamental shift in the demands placed on high-performance computing infrastructure. Data processing in areas such as artificial intelligence and big data has, in recent years, pushed traditional HPC systems well beyond their original design envelopes. Analytics frameworks like Spark process enormous volumes of data through massive input and output operations, and these I/O operations frequently become the dominant bottleneck in the entire pipeline. While researchers have made remarkable progress in speeding up computation through in-memory processing and distributed execution, the storage layer beneath these frameworks has often lagged behind. When thousands of tasks simultaneously read and write terabytes of intermediate results, the file system can become the slowest component in the cluster, leaving expensive processors idle while they wait for data to arrive.

Parallel file systems have long been the standard answer to this challenge in HPC environments. Systems such as Lustre stripe files across many storage servers so that multiple clients can read and write data simultaneously, and techniques such as file partitioning and replication provide both high performance and high availability. HDFS, meanwhile, was designed specifically for the Hadoop ecosystem, replicating blocks across commodity nodes to achieve fault tolerance and data locality. Yet both systems were conceived for workloads that differ in important ways from those generated by modern analytics engines. Spark, in particular, was designed around in-memory processing and resilient distributed datasets, and its I/O behavior, characterized by many small, concurrent, short-lived read and write operations, does not always align with the assumptions built into these general-purpose storage platforms.

This is where the concept of an ad-hoc parallel file system comes in. Rather than serving as a permanent, cluster-wide storage fabric, an ad-hoc file system is deployed on demand alongside a specific analytics job, using the same nodes that run the computation. Expand, the file system at the heart of the new study, is a parallel and distributed file system designed by the ARCOS research group that can operate in precisely this mode. By co-locating storage with computation, Expand eliminates much of the network traffic and metadata overhead that conventional file systems incur when data must be fetched from dedicated storage servers. The researchers describe this approach as a way to alleviate the I/O bottlenecks that arise in traditional parallel file systems when they are asked to serve big data analytics workloads.

The technical integration with Apache Spark is a central contribution of the work. Spark is written in the Java programming language and runs on the Java virtual machine, which means that a file system written in a language such as C or C++ cannot be called directly by the framework. The researchers addressed this by exposing Expand to Spark as a Hadoop-compatible file system, allowing the analytics engine to interact with Expand through the standard interfaces it already uses, with a Java native interface bridging the gap between the two software layers where needed. This design means that existing Spark applications can take advantage of Expand without being rewritten, preserving the benefits of both platforms: Spark’s powerful distributed computation model and Expand’s high-performance storage architecture.

Under the hood, Expand relies on the same principles that make modern parallel file systems effective, including file partitioning and replication, but it applies them with the specific access patterns of analytics workloads in mind. Data is distributed across the storage servers that participate in the ad-hoc deployment, so that read and write operations can proceed in parallel across many disks and network paths simultaneously. Replication ensures that the failure of a single node does not compromise the availability of the data during a job’s execution. The combination is intended to provide highly available, high-performance storage precisely when and where an analytics application needs it, and to be dismantled just as quickly when the job is done.

To evaluate the approach, the researchers compared Expand against two heavyweight competitors: Lustre, one of the most widely used parallel file systems in supercomputing centers around the world, and HDFS, the storage backbone of the Hadoop ecosystem. The evaluation used TeraSort, a benchmark that has become a de facto standard for measuring the end-to-end performance of big data platforms because it exercises sorting, shuffling, and both reading and writing of very large datasets. The results were striking. On this benchmark, Expand achieved performance up to 4.5 times higher than Lustre and up to 2.5 times higher than HDFS. These are not marginal improvements; in a field where a few percentage points of speedup can translate into significant cost savings, a several-fold improvement suggests that the storage layer, rather than the compute layer, was the dominant constraint in the tested configurations.

The implications of the study extend beyond a single benchmark. As artificial intelligence and big data applications continue to grow in scale and importance, the cost of I/O bottlenecks compounds across every stage of the data lifecycle, from ingestion and cleaning to model training and result aggregation. The finding that an ad-hoc file system, deployed only for the duration of a job and co-located with the computing resources, can substantially outperform permanently provisioned storage infrastructures suggests that analytics clusters could be re-architected around tighter integration of storage and computation. It also highlights a potential inefficiency in the common practice of running analytics frameworks on top of storage systems designed for other purposes, whether for traditional HPC simulation workloads in the case of Lustre or for batch processing in the case of HDFS.

The work forms part of a broader research effort at the ARCOS group to develop high-performance storage for hybrid HPC and data-intensive workloads. The research was partially supported by the Spanish Ministry of Science and Innovation through the project Expand: High Performance Storage System for HPC and Big Data Environments, funded under the European Union NextGenerationEU and PRTR programs, and through the SCIOT project on new scalable I/O techniques for hybrid workloads. The authors also acknowledged the use of the C3 computer resources provided by the Centro para el Análisis y Modelado de Sistemas Complejos en Ingeniería y Biomedicina. The study is open access, and the researchers report no competing interests.

For practitioners, the message of the research is that the storage layer of a big data analytics ecosystem is not a fixed constraint but a design choice with real performance consequences. As data volumes continue to explode across fields from genomics to climate modeling to commercial machine learning, solutions that treat storage as an integral, workload-aware component of the analytics pipeline, rather than a passive repository, may define the next generation of data platforms. The Madrid team’s results with Expand provide a concrete, benchmarked demonstration that rethinking the file system itself, and adapting it to the specific rhythms of frameworks like Apache Spark, can unlock performance that conventional architectures leave on the table.

Subject of Research: Ad-hoc parallel file systems for accelerating big data analytics workloads in Apache Spark environments.

Article Title: Improving big data analytics ecosystems using ad-hoc parallel file systems

Article References: Sotodosos-Morales, G., Garcia-Carballeira, F., Camarmas-Alonso, D., Calderon-Mateos, A., Muñoz-Muñoz, D., & Carretero, J. (2026). Improving big data analytics ecosystems using ad-hoc parallel file systems. Journal of Big Data. https://doi.org/10.1186/s40537-026-01559-6

Image Credits: AI Generated

DOI: 10.1186/s40537-026-01559-6

Keywords: big data analytics, parallel file system, Apache Spark, Expand, HDFS, Lustre, high-performance computing, I/O bottlenecks, TeraSort, distributed storage, in-memory processing, ad-hoc file system