LHCb Upgrade Streamlines Massive Data Processing for Run 3

The LHCb detector at the Large Hadron Collider has successfully implemented a comprehensive upgrade to its offline data processing and analysis systems in preparation for the third data-taking run, known as Run 3. This upgrade, designated as LHCb Upgrade I, addresses the significant challenges posed by a thirtyfold increase in data volume resulting from higher luminosity, larger average event sizes, and improved trigger efficiencies.

Central to this new infrastructure is the “Sprucing” process, a centralized offline data processing stage that skims, slims, and splits the raw data into multiple sub-streams. The Sprucing step is designed to reduce the total data rate from the initial 10 GB/s written to tape to a sustainable 3.5 GB/s for offline disk resources.

Complementing the Sprucing process is the “Analysis Productions” system, a highly automated and declarative framework for producing nTuples, which are rectangular datasets used for high-level physics analysis. In previous runs, individual analysts would submit user jobs to create these nTuples, a process that was labor-intensive and prone to errors. The new Analysis Productions system extends the LHCb Dirac transformation system to provide a simplified user interface for configuring and testing workflows on the World-wide LHC Computing Grid (WLCG).

The LHCb experiment collects data in three primary physics streams: FULL, TURBO, and TURCAL. Each stream is handled differently by the Sprucing process to optimize data reduction and preservation. The FULL stream contains inclusive selection lines, allowing for the re-processing of data with new selection lines in the future. The TURBO stream focuses on exclusive selection lines for specific decay channels, persisting only the necessary physics objects. The TURCAL stream includes raw detector information for calibration purposes.

In 2024, LHCb recorded a record 9.56 fb⁻¹ of data, demonstrating the effectiveness of the upgraded systems in handling such a large volume. The Sprucing process achieved bandwidth reductions of a factor of 7 for the FULL stream and a factor of 8 for the TURCAL stream. These reductions were critical in ensuring that the data could be stored and accessed efficiently. The successful implementation of these processes allowed analysts to begin analyzing data within 2–3 days of it being recorded, a significant improvement over previous runs and a first for any LHC experiment.

The DaVinci application plays a pivotal role in the Analysis Productions system, serving as the offline analysis software that creates nTuples from the Spruced data. Built on the Gaudi framework, DaVinci employs the FunTuple component to store event data in ROOT format, optimizing it for subsequent analysis. The application allows for the reconstruction of particle decays and the calculation of derived quantities, such as isolation and flavor tagging. By sharing algorithms and tools with the online trigger and Sprucing systems, DaVinci ensures consistency and efficiency in the data processing pipeline.

To further enhance the workflow, the LHCb team has developed the apd tool, a Python package that provides a simple interface for retrieving data from Analysis Productions. The apd tool allows analysts to look up physical file names based on metadata tags, eliminating the need to manage lists of time-limited file names. This tool integrates with the Snakemake workflow management system, enabling the tracking of dependencies between data artifacts and ensuring the reproducibility of analyses. The apd tool also facilitates the preservation of analyses by abstracting network connectivity, allowing for the substitution of data access methods as technology evolves.

In LHCb Upgrade II from Run 5 onwards, the experiment will increase its instantaneous luminosity by a factor of five, leading to an expected factor-of-five increase in nTuple data volume. To address this, LHCb is developing an extended Analysis Productions model that will move more of the data processing and filtering steps into centralized productions. This model will include the application of selection cuts, evaluation of derived quantities, and the use of machine learning models for inference. By centralizing these processes, LHCb aims to manage the data volumes effectively and ensure that the analysis remains efficient and scalable.

The extended Analysis Productions model will also involve the creation of “Analysis Trains,” where physics working groups combine their productions to run over the same input data. This approach reduces the data reading load on the WLCG sites and optimizes resource usage. The model is designed to be flexible, allowing analysts to prototype algorithms and train machine learning models on a subset of the data before applying them to the full dataset. This workflow is well-suited to future data storage and retrieval methods that may rely on performative data recall from tape. The LHCb team is working closely with the WLCG to guide its development, ensuring that the infrastructure can support the needs of the experiment in the coming decade.

Subject of Research: LHCb offline data processing and analysis systems for Run 3

Article Title: The LHCb Sprucing and Analysis Productions

Article References: Abdelmotteleb, A., Bertolin, A., Burr, C., Couturier, B., Eckstein, E., Fazzini, D., Grieser, N., Haen, C., O’Neil, R., Rodrigues, E., Skidmore, N., Smith, M., Wiederhold, A. R., & Zhang, S. (2025). The LHCb Sprucing and Analysis Productions. Computing and Software for Big Science, 9(1), Article 15. https://doi.org/10.1007/s41781-025-00144-5

Image Credits: AI Generated

DOI: 10.1007/s41781-025-00144-5

Keywords: LHCb, Large Hadron Collider, Data Processing, Run 3, Sprucing, Analysis Productions, DaVinci, High-Energy Physics, Data Management, LHC Upgrade I, Analysis, Productions