dCache Storage System Supports Global Scientific Data Management

dCache is a distributed storage system designed to manage the massive volumes of data generated by modern scientific facilities, including the European X-ray Free-electron Laser (EuXFEL) and the Large Hadron Collider (LHC). Developed through a collaboration between the Deutsches Elektronen-Synchrotron (DESY), Fermi National Accelerator Laboratory (Fermilab), and the Nordic e-Infrastructure Collaboration (NeIC), the system addresses the challenges of the entire data life cycle. This includes efficient data ingest, analysis, and long-term preservation, typically involving large tape libraries. The software provides a unified method to support various scientific use cases, such as high-throughput data ingest, wide-area data sharing, and access from high-performance computing (HPC) clusters.

Initially created for high energy physics (HEP) experiments, dCache has evolved into a tool used by diverse scientific communities, including astrophysics, biomedical research, and life sciences. The system is written in the Java programming language, allowing it to run on various operating systems. Its architecture strictly separates the file namespace from the physical location of the data, a design choice that enables location transparency. This means that a file’s name does not change when its physical location is moved, allowing users to access the same file from different hosts without modification.

The core architecture of dCache consists of four main components: doors, pools, the pool manager, and the namespace. Doors serve as user entry points implementing specific access protocols, while pools are the data servers that store the actual data. The pool manager is responsible for data placement, selecting which pool should handle a given transfer based on configurable policies. The namespace component stores file metadata and enforces POSIX semantics. These components communicate over a TCP/IP network, and the system scales horizontally by adding new data servers, which increases both storage capacity and aggregated data bandwidth.

As of March 2025, dCache instances account for more than 50% of the overall storage capacity in the Worldwide Large Hadron Collider Computing Grid (WLCG), excluding CERN. This significant share underscores the system’s role in underpinning major scientific discoveries, including the Higgs boson. The system has been in production for over 20 years and is deployed globally. For instance, the Photon Science instance at DESY and the public dCache at Fermilab each host more than 1.5 billion files, demonstrating the scalability of the namespace service. The EuXFEL instance at DESY combines more than 400 physical hosts to provide 120 petabytes of total capacity.

dCache supports federated systems, allowing different institutes to contribute resources to an aggregated storage system spanning countries. Two WLCG sites currently operate in such distributed deployments: the Nordic Data-Grid Facility (NDGF) Tier-1 and the ATLAS Great Lake Tier-2 (AGLT2). NDGF is a single distributed Tier-1 center formed by universities from Denmark, Sweden, Norway, Finland, Slovenia, and Switzerland. Core services run in Copenhagen, while storage nodes are hosted by the participating institutes. This setup ensures that the Tier-1 center can accept data from CERN even if some sites are unavailable, providing high availability and fault tolerance.

To manage data efficiently, dCache introduces a concept of quality of service (QoS) for storage, developed as part of the INDIGO-DataCloud project. QoS describes how data is stored and is exposed as a REST API, extending the long-standing support for storing data on disk and tape. This mechanism allows users to specify storage preferences based on latency, reliability, and cost constraints. The Resilience subsystem, responsible for data durability, is evolving into the QoS engine to address these requirements. This evolution is particularly important for scientific communities like EuXFEL, where durability and availability play a critical role in the data life cycle.

The system includes a Bulk Service that enables users to perform operations on a large number of files, such as deletions, data migrations to or from tape, and QoS transitions. This service is built as an asynchronous task execution module that allows users to submit batch requests via a RESTful API. The dCache Bulk Service API has been adopted by WLCG storage providers as the WLCG Tape API, replacing the older SRM interface as the standard protocol for interaction with tape storage. This standardization facilitates efficient data movement and management across the global grid.

For non-HEP analysis, dCache supports access via NFSv4.1/pNFS, allowing the storage system to be mounted as a locally mounted filesystem without requiring special software on worker nodes. This is crucial for using standard tools like Jupyter Notebooks and Apache Spark. The system also supports the SMB protocol for Microsoft Windows users, implemented via a protocol translation server running the open-source SAMBA software. Additionally, dCache is an early adopter of NFS-over-TLS, working with Linux kernel developers to provide secure data access. These protocol-agnostic features ensure wide interoperability across different scientific computing environments.

Magnetic tape remains the most economical solution for long-term data archival, particularly for rarely accessed cold data. Tape cartridges can store up to 50 terabytes of uncompressed data and have an expected lifetime of 15 to 30 years. However, tape systems have downsides, including a long latency of about 50 seconds for tape positioning and mounting times that can reach minutes. To address the inefficiency of storing small files on tape, dCache has deployed a small file service at DESY since 2015. This service packs small files into containers based on specified aggregation policies, improving tape resource utilization and scalability.

dCache integrates with the CERN Tape Archive (CTA), an open-source tape storage system designed to replace the legacy CASTOR system. The CTA project is adding flexibility to allow wider adoption by other sites. dCache developers at DESY have implemented a CTA-specific nearline storage provider called dcache-cta, which communicates with the CTA frontend using Google’s gRPC library. This integration allows dCache to submit archive and retrieve requests directly to the CTA scheduler, optimizing data migration between disk and tape media. The system also supports storage events, which are asynchronous notifications generated when specific actions occur, such as file creation or migration, allowing for real-time workflow automation.

The development process of dCache emphasizes software quality and build reproducibility. The project has adopted semantic versioning and a time-based release policy, using continuous integration systems to execute automatic build and test procedures. The build and test pipeline is described in a YAML-formatted file and consists of multiple stages, including compiling code, running unit tests, and validating deployment on target platforms. The pipeline relies on Linux containers and Kubernetes for orchestration, allowing multiple versions of dCache components to be deployed simultaneously to test backward compatibility. This rigorous testing ensures that the software remains stable and reliable for critical scientific infrastructure.

Future work for dCache focuses on preparing for upcoming challenges expected by the EuXFEL upgrade, the High-Luminosity LHC, and PETRA-IV. Developers are working on providing an easy-to-deploy container-based solution for federated systems, which can be pre-configured by the main site and distributed to satellite sites. This approach aims to reduce operational overhead and improve data availability. Additionally, the project is integrating a Prometheus exporter for monitoring and is exploring machine learning methods to detect data access anomalies automatically. These efforts will help the system adapt to changing user access patterns and hardware behavior, ensuring that dCache remains a flexible and future-proof storage solution for managing petabyte-scale datasets efficiently.

The dCache architecture relies on a namespace component that manages file metadata through a layered design. The topmost layer, PnfsManager, handles interactions between the rest of the system and the underlying filesystem backend. Below this sits a filesystem abstraction layer, currently implemented as Chimera, which is built on top of a relational database. In production deployments, dCache uses the PostgreSQL database, while development and unit testing rely on the embedded HSQLDB system. This structure allows for gradual evolutionary changes to refine the namespace component in response to new requirements or technology shifts. An integrated automatic database schema migration applies filesystem structural changes when new versions are installed and enables software downgrades if necessary.

Performance improvements in the metadata service have yielded measurable results in file creation rates. With improvements introduced in version 9.2, dCache demonstrated a file-creation rate of up to 17,500 files per second when running the widely used HPC benchmark mdtest. This benchmark specifically creates zero-byte files to isolate and measure namespace latency and throughput, rather than testing data transfer speeds. The ability to handle such high rates of metadata operations is critical for workflows that involve the creation and management of large numbers of small files, a common scenario in photon science and other data-intensive fields.

Data distribution within dCache is managed by the PoolManager service, which decides which data server will handle transfer requests based on configurable policies. The system supports several placement policies, or partition types, including Classic, LRU, Random, and WASS. The WASS policy, which stands for weighted available space selection, is the default partition type. It selects pools randomly weighted by available space, incorporating factors such as the time since a file was last accessed, the amount of garbage collectible files, and current load information. This approach aims to provide a balanced distribution of load across the available storage nodes, preventing hot spots and ensuring efficient resource utilization.

For clients accessing data, dCache employs a redirect mechanism to scale with the number of users. When a client requests a file, the door component queries the metadata server for file attributes and location. After checking access rights, the door asks the PoolManager for a suitable pool. The selected pool then returns client redirect information to the door, which passes it to the client. The client subsequently connects directly to the pool to perform I/O requests. This flow includes a built-in self-consistency correction; if a pool receives a request for a file replica it does not host, the metadata server is notified, the inconsistent record is removed, and the selection process restarts to ensure data integrity and availability.

Subject of Research: Distributed storage system architecture and deployment for data-intensive scientific research

Article Title: dCache: The Storage System of Choice for Data-Intensive Applications

Article References: Mkrtchyan, T., Green, C., Hoyos, K., Litvintsev, D., Millar, P., Morschel, L., Rossi, A., Sahakyan, M., & Starr, D. (2025). dCache: The Storage System of Choice for Data-Intensive Applications. Computing and Software for Big Science, 9(1), Article 20. https://doi.org/10.1007/s41781-025-00152-5

Image Credits: AI Generated

DOI: 10.1007/s41781-025-00152-5

Keywords: dCache, Data Management, High Energy Physics, Distributed Storage, WLCG, Tape Archive, Scientific Computing, DESY, Fermilab, Data Preservation, Storage, System