New Co-Design Framework Speeds Up LLM Inference on Mixed Hardware by Taming Memory Bottlenecks

Large language models have transformed how the world writes, codes, and searches, but behind every fluent answer lies a punishing engineering problem: inference, the act of actually running these models, remains slow, expensive, and dominated by hardware limits rather than raw intelligence. A research team from Shandong University of Science and Technology and Huazhong University of Science and Technology has now published a framework called LLM-CoOpt that attacks this problem from the algorithm and hardware sides simultaneously, reporting measurable gains in throughput and latency without sacrificing model accuracy. The work, described in the journal Cluster Computing, targets the three bottlenecks that most often throttle large language model serving: memory bandwidth saturation, computational redundancy, and the growing pain of processing very long input sequences.

The central insight behind LLM-CoOpt is that software optimizations and hardware realities cannot be designed in isolation. Modern language model inference runs on heterogeneous platforms, machines that combine different types of processors and accelerators, each with its own memory hierarchy, compute throughput, and bandwidth characteristics. On such systems, a clever algorithm that ignores how data actually moves through the hardware can end up slower than a naive one. The researchers therefore built their framework as a co-design effort, shaping each optimization around the memory access patterns and compute structures of the underlying platform rather than treating the model as an abstract mathematical object.

The first pillar of the framework, called Opt-KV, addresses what many engineers consider the single most stubborn constraint in transformer inference: the key-value cache. When a language model generates text, it stores intermediate representations, known as the key-value cache, for every token it has processed so that future tokens can attend to the full context without recomputing everything. As conversations and documents grow longer, this cache balloons in size, and reading and writing it repeatedly pushes inference squarely into a memory bandwidth bottleneck, a phenomenon researchers have dubbed the memory wall. Opt-KV improves memory access efficiency by optimizing both the write path and the read path of the cache, restructuring how data flows in and out of memory so that each byte moved does more useful work.

On top of that memory traffic reduction, Opt-KV introduces FP8 quantization for the cache. Quantization compresses numerical values from their usual high-precision formats into fewer bits, shrinking the memory footprint of the cache and easing pressure on the bandwidth that connects compute units to storage. FP8, an eight-bit floating-point format developed specifically for deep learning workloads, preserves a dynamic range that pure integer formats lack, which helps maintain accuracy while still halving or better the memory cost compared with sixteen-bit representations. The team reports that this combination maintains model accuracy while meaningfully reducing the memory the cache demands, a trade-off that has historically been difficult to strike because aggressive cache compression tends to degrade the quality of generated text.

The second pillar, Opt-GQA, targets computational redundancy inside the attention mechanism itself. Standard multi-head self-attention, the operation that lets a model weigh the relevance of every previous token when processing a new one, computes separate key and value projections for every attention head. That design multiplies memory traffic and computation, even though many heads end up learning similar key-value patterns. Opt-GQA restructures multi-head self-attention into grouped-query attention, a scheme in which multiple query heads share a single set of key and value projections. By grouping queries this way, the framework cuts the overall computational complexity of attention, which translates directly into higher throughput and lower resource consumption on the target hardware.

Grouped-query attention is not an entirely new idea, but the framework’s contribution lies in integrating it into a co-designed serving pipeline where its benefits compound with the memory optimizations of Opt-KV. Because attention reads the key-value cache on every generated token, reducing the number of distinct key-value projections also reduces the volume of cache data that must be streamed during decoding. The two optimizations therefore reinforce each other: less cache to move, and fewer operations that need to touch it. This kind of interaction is precisely why the authors argue for algorithm-hardware co-design rather than applying optimizations one at a time and hoping their benefits add up.

The third pillar, Opt-Pa, confronts the challenge of long-sequence processing. Long-context inputs, such as entire codebases, lengthy legal documents, or extended chat histories, stress inference systems in a qualitatively different way than short prompts. The key-value cache for tens of thousands of tokens can exceed what fits comfortably in fast memory, forcing evictions, fragmentation, and stalls. Opt-Pa adopts a two-step strategy: it first segments long sequences into manageable chunks, and then applies lazy memory mapping and computation to those chunks. Lazy mapping means memory is allocated and computed only when actually needed rather than eagerly up front, which significantly reduces memory pressure and improves performance on long-context inputs. The approach builds on the paged attention concept popularized by the vLLM serving system, which treats cache memory like the virtual memory pages of an operating system, but extends it with the chunking and lazy evaluation strategy tailored for heterogeneous platforms.

To validate the framework, the researchers ran experiments on the LLaMa-13B-GPTQ model, a thirteen-billion-parameter language model already compressed through GPTQ weight quantization, making it a realistic candidate for deployment on resource-constrained serving hardware. The results show that LLM-CoOpt increases inference throughput by up to 13.43 percent and reduces latency by up to 16.79 percent, all while maintaining model accuracy. Those figures may sound modest compared with headline-grabbing claims of order-of-magnitude speedups, but they represent gains achieved without retraining the model, without changing its outputs, and without requiring exotic hardware, which is exactly the profile that matters for production inference clusters where reliability and fidelity are non-negotiable.

The evaluation drew on publicly available datasets and models, including the ARC reasoning benchmark from the Allen Institute for AI and the ShareGPT V3 conversational dataset, alongside Meta’s LLaMA models. The team also acknowledged the open-source tooling that underpins the work, including vLLM and GPTQ, underscoring how much of modern inference research is built on a shared foundation of open systems. The research was supported in part by the National Natural Science Foundation of China, the ghFund, and the Shandong Provincial Natural Science Foundation. Jie Kong performed the experiments and drafted the manuscript, Wei Wang assisted with experiments and preparation, and Jiehan Zhou and Chen Yu supervised the research, with all authors reviewing the final paper.

The broader significance of LLM-CoOpt lies in what it says about the future of AI infrastructure. As language models are embedded into search engines, coding assistants, and enterprise tools, the cost of serving them, not the cost of training them, increasingly dominates the economics of AI. Every percentage point of throughput gained on existing hardware translates into real savings and lower latency for users, and techniques that preserve accuracy while trimming memory traffic extend the useful life of deployed accelerators. By demonstrating that cache optimization, attention restructuring, and paged long-sequence handling can be woven together into a single co-designed framework, the researchers offer a practical, high-performance path for real-world inference of large-scale language models, one that treats the memory wall not as an immovable barrier but as an engineering problem with room left to solve.

Subject of Research: Algorithm-hardware co-design framework for efficient large language model inference on heterogeneous computing platforms

Article Title: LLM-CoOpt: a co-design and optimization framework for efficient LLM inference on heterogeneous platforms

Article References: Kong, J., Wang, W., Zhou, J., & Yu, C. (2026). LLM-CoOpt: a co-design and optimization framework for efficient LLM inference on heterogeneous platforms. Cluster Computing, 29(15), Article 861. https://doi.org/10.1007/s10586-026-06711-8

Image Credits: AI Generated

DOI: 10.1007/s10586-026-06711-8

Keywords: large language models, LLM inference, KV cache optimization, FP8 quantization, grouped-query attention, paged attention, heterogeneous platforms, memory bandwidth, vLLM, LLaMA, throughput optimization, algorithm-hardware co-design