MIT Physics Analysis Facility Demonstrates High-Speed Computing for Future Particle Colliders

Physicists at the Massachusetts Institute of Technology have deployed a specialized computing platform named SubMIT to investigate advanced concepts for processing and analyzing scientific data from high-energy particle physics experiments. Developed within the MIT Department of Physics, the SubMIT facility functions as a prototype Analysis Facility designed to address the unprecedented computational challenges anticipated during the upcoming High-Luminosity phase of the Large Hadron Collider at CERN. Anticipated to commence operations within the next decade, the High-Luminosity Large Hadron Collider is projected to produce scientific datasets reaching the multi-exabyte threshold, exceeding the processing power and storage capacities possible under the current worldwide computing model even after factoring in expected hardware improvements. In response to these future big data demands, which are mirrored in upcoming experimental efforts such as the Deep Underground Neutrino Experiment, the Large Synoptic Survey Telescope, the Square Kilometer Array, and the Electron-Ion Collider, the SubMIT project explores modern architectural solutions that decouple end users from low-level computing logistics while accelerating turnaround times for exploratory data analysis across departmental and collaborative research programs.

The traditional computing model employed by the Worldwide LHC Computing Grid relies on a multi-tiered architecture that distributes data and simulation processing across independent computing nodes around the globe. While this distributed structure has proven successful for handling large-scale data production and standardized reconstruction workflows over the past two decades, it presents significant friction when adapted to iterative end-user analysis. In conventional grid environments, researchers must manage complex job submission protocols, configure authentication certificates, navigate long batch queue latencies, and operate without direct interactive access to computational resources co-located with their data files. To eliminate these barriers, the High Energy Physics community has identified a critical need for modern analysis facilities that facilitate direct data exploration. Emerging analytical frameworks, including ROOT RDataFrame, Coffea, Uproot, and Awkward Array, leverage columnar data formats such as NanoAOD at CMS and PHYSLITE at ATLAS. By processing events in declarative pipelines without requiring sequential loops or intermediate data merging steps, these tools maximize parallel efficiency but require underlying hardware platforms optimized for high-bandwidth data access, low latency, and rapid interactive scaling.

To support this operational paradigm, the SubMIT facility couples high-speed local processing with dedicated hardware resources optimized for diverse computational tasks. Interactive access is provided through a pool of eight dedicated login hosts that allow users to develop, test, and debug their analytical code in real time before scaling workloads to distributed clusters. The platform integrates high multicore processor architectures, including dual AMD EPYC 9654 processors offering 96 physical cores and 384 execution threads through Simultaneous Multithreading, alongside dual AMD EPYC 9965 configurations delivering 192 cores and 768 threads with 1.5 terabytes of main system memory. These machines are engineered specifically for memory-intensive analytical operations that benefit from native multithreading over coarse-grained distributed task splitting. In addition to high-density central processors, the infrastructure incorporates nineteen accelerator nodes hosting 66 dedicated graphics processing units, which include two NVIDIA RTX A6000 cards, twelve V100 cards, eight A30 cards, and 44 GeForce GTX 1080 cards, enabling researchers to conduct machine learning development, train neural networks, and execute automated support services within a unified hardware environment.

Efficient data movement is maintained across SubMIT through an optimized storage hierarchy interconnected by dedicated network interfaces delivering a minimum throughput of 100 gigabits per second per server node. Upon account creation, each registered user receives an initial baseline storage allocation comprising a 10-gigabyte redundant disk array for home directories and notebook scripts, a 100-gigabyte quota for local software installations and environments, and a 1-terabyte volume designated for individual research data collections. For large-scale data handling, the facility incorporates a centralized 100-terabyte solid-state scratch volume configured on high-performance Non-Volatile Memory Express drives, alongside a 1.6-petabyte distributed file system built upon CephFS spinning disks for mass group storage. The platform also interfaces directly with external mass storage systems, leveraging 2 petabytes of dedicated analysis disk storage hosted on a CephFS file system at the MIT Tier-2 center, which maintains 15 petabytes of total capacity. To accelerate repeated data reads, SubMIT integrates an external XCache proxy server located at the ESnet facility in Cambridge, which caches frequently requested datasets on local solid-state drives.

The facility provides an accessible, secure interface that accommodates varying user backgrounds through both remote command-line sessions and browser-based access portals. Secure terminal access is established via SSH using uploaded public keys, backed by MIT Lightweight Directory Access Protocol authentication and mandatory two-factor verification. For graphical workflows, the team deployed a centrally managed JupyterHub instance integrated with MIT Shibboleth single sign-on authentication, which allows analysts to spawn personal computational servers across SubMIT cluster nodes via the Slurm workload manager. Software flexibility and reproducibility are achieved through extensive containerization support utilizing Apptainer and Podman, enabling users to execute custom user-space runtime environments without compromising host security. Furthermore, centrally validated software frameworks and shared CERN software distributions are mounted dynamically across all cluster nodes using the CernVM-File System. The platform has demonstrated operational resilience through major infrastructure transitions, successfully completing migrations from the CentOS 7 operating system to AlmaLinux 9 and transitioning underlying shared storage architectures from GlusterFS to CephFS using staged test partitions and legacy runtime containers.

Workload orchestration within SubMIT is partitioned into distinct tiers to balance low-latency local execution with massive distributed scale-out capabilities. For medium-sized computational jobs and localized task execution, SubMIT maintains a dedicated internal cluster comprising approximately 2,500 processor cores coupled with 2 gigabytes of random-access memory per execution core, scheduled directly through the Slurm job management system. Distributed parallel frameworks, including Dask Gateway and OpenMPI, interface natively with Slurm to dynamically allocate worker processes across cluster nodes, streamlining parallel execution directly through application programming interfaces. When analytical tasks exceed local cluster boundaries, the facility routes workflows to external distributed computing infrastructures through an integrated HTCondor scheduler. This setup connects users to the MIT Tier-2 facility with 25,000 processor cores, the MIT Tier-3 center with 700 processor cores, the global CMS computing pool, the Open Science Grid, and specialized departmental resources such as the MIT Earth, Atmospheric and Planetary Sciences cluster and the local Lattice Quantum Chromodynamics cluster, using XRootD protocols to access remote datasets.

Demographic evaluations conducted between late 2024 and early 2025 illustrate that SubMIT supports a diverse academic and scientific user base across multiple institutions and disciplines. Out of approximately 1,000 registered accounts, roughly 800 users utilize the facility for academic coursework, while approximately 200 active research users access the cluster consistently on a monthly basis. Among research users, approximately 25 percent represent external collaborators working with MIT investigators, spanning high-energy physics, astrophysics, nuclear science, theoretical physics, electrical engineering, and aerospace engineering. Usage analytics show that 73 percent of research users interact with the system interactively through SSH terminal sessions or JupyterHub notebook servers, while 27 percent dispatch batch processing jobs. Furthermore, 75 percent of active workloads rely on standard processor cores, while 25 percent utilize graphics processing units. To maintain smooth daily operations, user support combines extensive documentation, a help desk staffed by active early-career physicists, and an automated conversational assistant called A2rchi, which uses retrieval-augmented generation and large language models to assist with troubleshooting and support ticket triage.

Rigorous benchmark evaluations have been carried out on SubMIT using the Analysis Grand Challenge framework, a standardized top quark pair production analysis using 1.7 terabytes of open experimental data collected by the CMS collaboration. Because typical analysis tasks read only approximately 5 percent of total input datasets, read benchmarks were executed on a dedicated 192-core server to evaluate input-output throughput without processing bottlenecks. The experimental trials confirmed that local solid-state drives operating over 100-gigabit network links achieved the highest data transfer rates, followed by the NVMe-backed ESnet XCache service and remote XRootD streams from CERN-EOS and Fermilab-EOS storage. Conversely, spinning-disk CephFS configurations exhibited higher latency due to fixed 16-megabyte read-ahead buffer requirements that exceeded the granular read sizes requested by ROOT columnar operations. Scalability benchmarks executing RDataFrame via Dask Gateway and Slurm revealed that distributing tasks across multiple independent worker nodes introduced negligible scheduling overhead compared to multi-process execution on a single physical host, verifying the efficiency of the platform for distributed high-throughput processing.

Beyond benchmark workflows, SubMIT has validated its operational model by supporting four complete scientific measurements carried out by CMS physicists from initial design to peer-reviewed publication. These include a Standard Model measurement of opposite-sign W boson pair production cross-sections, a high-precision measurement of the W boson mass, searches for rare Higgs boson decays, and an exotic search for Soft Unclustered Energy Patterns. While standard analyses relied directly on centrally generated NanoAOD files, complex studies required specialized custom NanoAOD datasets containing detailed tracking and particle flow information produced across grid resources. In the precision W boson mass analysis, researchers processed custom event data to generate massive multidimensional histograms reaching 20,000,000 bins for systematic uncertainty estimations, utilizing Boost histograms with atomic C++ aggregation saved into HDF5 files. Complex statistical fits were accelerated using CombineTF, an analytical framework leveraging TensorFlow automatic differentiation and multithreaded minimization to achieve substantial execution speedups compared to traditional fitting tools, confirming the benefits of heterogeneous computing and high-speed data storage.

The flexibility of the SubMIT platform extends beyond Large Hadron Collider operations to future collider feasibility investigations and university educational instruction. The system supports ongoing design studies for the proposed Future Circular Collider electron-positron stage, hosting 20 terabytes of simulated events in the EDM4hep data format within Key4hep software environments, allowing researchers to evaluate Higgs, W, and Z boson physics alongside detector response simulations. Concurrently, SubMIT serves as an educational computing infrastructure across introductory and advanced physics courses at MIT. Large-enrollment courses structured around the Technology-Enabled Active Learning format utilize the cluster to deliver course-specific artificial intelligence tutoring and run simulations of electromagnetic fields, relying on pre-scheduled Slurm compute reservations to guarantee immediate capacity during classroom sessions. Advanced undergraduate and graduate astrophysics courses leverage CVMFS-distributed Conda environments containing AstroPy and Gadget cosmological simulation packages executed via OpenMPI. By combining interactive responsiveness, heterogeneous hardware, and accessible software deployment, SubMIT demonstrates a practical, scalable blueprint for scientific computing facilities facing the computational frontiers of modern physics.

Subject of Research: Interactive computing infrastructure and analysis facility architecture for high-energy physics research

Article Title: SubMIT: A Physics Analysis Facility at MIT

Article References: Acosta, P., Bendavid, J., D’Alfonso, M., Eysermans, J., Freer, C., Goncharov, M., Heine, M., Lavezzo, L., Newman, D., Moore, M., Paus, C., Shen, X., Walter, D., & Wang, Z. (2026). SubMIT: A Physics Analysis Facility at MIT. EPJ Research Infrastructures, 10(1), Article 2. https://doi.org/10.1007/s41781-025-00156-1

Image Credits: AI Generated

DOI: 10.1007/s41781-025-00156-1

Keywords: High-Energy Physics, Particle Physics, Analysis Facility, Computing Infrastructure, High-Luminosity Large Hadron Collider, Distributed Computing, MIT Physics, CERN, SubMIT, Physics, Analysis, Facility