How Data Centers Are Reengineering AI Training for Larger Language Models

Training a modern large language model is no longer simply a matter of assigning a neural network to a powerful computer. It is an extended, tightly coordinated operation involving thousands of accelerators, high-speed links, distributed storage, scheduling software and recovery systems. A survey by Jiangfei Duan, Shuo Zhang and colleagues maps the engineering behind that operation, showing how researchers are redesigning nearly every layer of the computing stack to keep models moving forward. The review focuses on three connected demands: scalability, efficiency and reliability. Together, these demands define whether a distributed training system can expand to very large clusters, use its hardware productively and survive the failures that become increasingly likely during jobs lasting weeks or months.

The scale of the challenge is illustrated by the training of LLaMA-3, which the survey reports took about 54 days using 16,000 H100-80GB GPUs on Meta’s production cluster. Even at that scale, the system reached a Model FLOPs Utilization, or MFU, of only about 38 to 41 percent. MFU measures how effectively available floating-point computing capacity is used during training; the unused portion can reflect communication delays, memory bottlenecks, synchronization, uneven workloads or inefficient kernels. The figures make clear why simply adding more processors does not automatically produce proportional gains. Every accelerator must receive data, exchange intermediate results or gradients and remain synchronized with its peers. A slow link, overloaded storage service or failed device can leave many other processors waiting.

Most of the models covered by the review use decoder-only Transformer architectures. Text is converted into tokens and then vectors, supplied with positional information and processed through repeated layers containing attention and feed-forward blocks. Attention calculates relationships between tokens by transforming them into query, key and value tensors and applying a weighted operation based on their similarities. In its conventional form, self-attention has computational and memory demands that grow quadratically with sequence length, making long contexts especially expensive. Architectural changes such as multi-query attention, grouped-query attention and multi-latent attention reduce some key-value storage pressures, while Mixture-of-Experts designs activate only a subset of feed-forward experts for each input. These model-level choices directly affect the hardware, communication patterns and memory strategies required for training.

The survey describes distributed training as a combination of complementary forms of parallelism. Data parallelism assigns different portions of a batch to different devices, then uses collective operations to aggregate gradients. Tensor parallelism divides the matrices inside individual layers and is generally best suited to tightly connected accelerators because intermediate activations must be exchanged frequently. Pipeline parallelism places consecutive groups of layers on different devices and passes activations between stages, making it useful across nodes with lower bandwidth but introducing idle “pipeline bubbles” while stages fill and drain. Sequence parallelism divides long token sequences among devices, reducing per-device activation and attention costs. In practice, these methods are combined into hybrid plans, often called three-dimensional parallelism when data, tensor and pipeline parallelism are used together.

Choosing that combination is a difficult systems problem because every benefit carries a cost. Fully replicated data parallelism is comparatively simple but stores duplicate model states on every device. Fully sharded approaches such as ZeRO-3 and Fully Sharded Data Parallelism distribute parameters, gradients and optimizer states across devices, sharply reducing memory use while increasing communication. Hybrid sharding provides a middle ground by replicating states within smaller groups and sharding them across those groups. Pipeline schedules attempt to reduce idle time by interleaving forward and backward computations, but more active micro-batches can create memory imbalance between stages. Automated parallelism systems address the complexity by searching possible partitions, estimating execution and communication costs and selecting a plan for a particular model and cluster. The review highlights approaches based on dynamic programming, simulators, reinforcement learning, Monte Carlo search and constraint-guided planning.

Hardware and networks form the physical foundation of these strategies. GPUs are well matched to the matrix and vector operations that dominate Transformer training and increasingly support mixed numerical formats, including FP16, BF16 and FP8. Their high-bandwidth memory and specialized tensor-processing units allow large matrix operations to run in parallel, while interconnects such as NVLink and NVSwitch provide faster communication than conventional PCI Express within a server. Other accelerator ecosystems, including AMD GPUs, Google TPUs, Intel Gaudi processors, Graphcore IPUs and Cerebras wafer-scale systems, offer different combinations of memory, compute capacity and software support. The survey emphasizes that software portability remains important: a strategy optimized for one accelerator and programming environment may require substantial adaptation to run efficiently on another.

Communication can become the dominant expense in a distributed job. Gradient synchronization creates periodic bursts of what network engineers call elephant flows, while tensor and expert parallelism can generate intensive all-to-all exchanges. Remote Direct Memory Access allows one machine to access memory on another without involving the operating system, and GPU-direct variants can move data between GPUs across nodes while bypassing the CPU. InfiniBand provides a dedicated high-performance fabric, whereas RDMA over Converged Ethernet brings similar capabilities to Ethernet-based data centers. Network layouts are increasingly designed around training traffic rather than general-purpose workloads. Rail-optimized architectures group corresponding GPUs through carefully arranged switches, while other designs use sparse connectivity, optical circuit switching or topology-aware routing. Load-balancing methods split large transfers across multiple paths, and congestion-control schemes regulate traffic to limit stalls and packet loss.

Storage must also be treated as part of the training engine. A 70-billion-parameter model can produce a checkpoint of about 980 gigabytes, according to the survey, and thousands of accelerators may need to save or reload such states together. Distributed file systems and object stores therefore need high write and read bandwidth as well as consistency, availability, protection and security. Training data presents a different challenge: LLaMA 3 was trained on more than 15 trillion tokens, while crawling, filtering and preparing web-scale data can involve volumes far larger than the final dataset. Caches such as Alluxio and JuiceFS can prefetch data from slower storage, keeping accelerators supplied even when each token is normally read only once. Cluster schedulers must coordinate these demands with GPU, CPU, memory and network resources, while balancing fairness, utilization, job packing, adaptive scaling and energy consumption across multiple users and workloads.

Memory optimization is central because training stores far more than the model’s visible parameters. Mixed-precision training requires parameters and gradients, higher-precision copies for stable optimizer updates, momentum and variance states, activations from the forward pass, temporary communication buffers and allocator space. For a model with Phi parameters, the survey estimates that model states alone can require 16Phi bytes under a common mixed-precision arrangement before activations and other buffers are counted. Activation recomputation reduces the peak by discarding selected intermediate tensors during the forward pass and rebuilding them during backpropagation, trading memory for additional computation. Sharding removes duplicate states, while CPU and NVMe offloading supplements limited GPU memory. Defragmentation techniques use improved allocation policies or virtual-memory stitching to turn scattered free regions into usable capacity. The goal is not merely to fit a model, but to prevent memory management from idling the entire cluster.

At the computation level, the review points to a shift from treating training as a sequence of generic operators toward designing complete dataflows for specific hardware. FlashAttention reduces high-bandwidth-memory traffic by processing attention in tiles, retaining intermediate results in faster on-chip memory and fusing matrix multiplication, softmax and related operations into efficient kernels without changing the exact attention result. Later variants improve parallel scheduling, asynchronous data movement and low-precision execution. Compilers such as TVM, Triton and TorchInductor can generate kernels, fuse operators and exploit memory locality across broader graphs, reducing the need for every optimization to be hand-written. Mixed precision extends the same principle: BF16 and FP16 are widely used, while FP8, INT8, INT4 and even one-bit or ternary representations are being investigated. These formats can reduce computation, storage and communication, but they require scaling methods, higher-precision accumulators or latent weights to prevent numerical errors from undermining training.

The reliability problem grows with both cluster size and training duration. A failure may come from a GPU, a network link, storage, software or a less obvious straggler that slows progress without producing an immediate crash. Because distributed training is often synchronous, a single stalled participant can leave thousands of others idle. Checkpointing provides a recovery point, but writing and restoring enormous states can itself interrupt useful work. The survey therefore treats anomaly detection, rapid fault diagnosis, resilient communication and automatic recovery as core components rather than afterthoughts. It also distinguishes the needs of pretraining from those of later alignment and reinforcement learning, where actor, critic, reward and reference models may alternate between inference and optimization. Newer asynchronous systems stream generated trajectories to trainers instead of waiting for the slowest rollout, while agentic training adds tool calls, sandboxes, verifiers and long-horizon interactions. The review’s broad conclusion is that future progress will depend on co-design: accelerators, memory, networks, storage, schedulers, compilers and learning algorithms must be optimized as one evolving system, with optical computing and optical networks among the possible technologies for overcoming the limits of conventional digital hardware.

Subject of Research: Distributed infrastructure for efficient large language model training

Article Title: Efficient training of large language models on distributed infrastructures: a survey

Article References: Duan, J., Zhang, S., Wang, Z., Jiang, L., Qu, W., Hu, Q., Wang, G., Weng, Q., Yan, H., Zhang, X., Qiu, X., Lin, D., Wen, Y., Jin, X., Zhang, T., & Sun, P. (2026). Efficient training of large language models on distributed infrastructures: a survey. Vicinagearth, 3(1), Article 9. https://doi.org/10.1007/s44336-026-00038-z

Image Credits: AI Generated

DOI: 10.1007/s44336-026-00038-z

Keywords: large language models, distributed training, GPU clusters, parallelism, AI accelerators, high-performance networking, memory optimization, mixed precision, fault tolerance, Efficient, training, large