The research, conducted by Fuli Qiao and Mehrdad Mahdavi of the Department of Computer Science and Engineering at Penn State, builds on Low-Rank Adaptation, or LoRA, one of the most widely used parameter-efficient fine-tuning techniques in modern artificial intelligence. Instead of retraining the billions of weights inside a large language model, LoRA freezes the original weights and injects small pairs of low-rank matrices into each layer. Only these small matrices are updated during training, which slashes memory and computation costs by orders of magnitude. The catch, the authors note, is that conventional LoRA fixes the rank to the same value across every layer and every task, a one-size-fits-all choice that leaves a crucial question unexplored: how much capacity does each layer actually need for each new task?

Qiao and Mahdavi’s answer is a method that treats rank as a dynamic resource rather than a fixed hyperparameter. Their approach, which the team calls CL-Rank, relies on a subspace similarity metric to measure how orthogonal — that is, how non-overlapping — the low-rank subspaces occupied by different tasks are within a given layer. When a new task arrives, the algorithm evaluates the geometry of the learned subspaces and adaptively increases the layerwise rank for the new task where overlap threatens to interfere with previously learned representations. By steering new knowledge into directions of parameter space that are roughly orthogonal to old knowledge, the method minimizes interference while still giving each new task enough expressive capacity to generalize well.

The underlying intuition echoes a classical idea from neuroscience and machine learning known as the stability–plasticity dilemma. A learning system must be plastic enough to absorb new information but stable enough to retain old skills. Since the pioneering work on catastrophic interference in connectionist networks in the late 1980s and the advent of modern continual learning benchmarks, researchers have tried an arsenal of remedies: replaying stored examples, regularizing important weights, growing new network branches, or constraining gradient updates to null spaces of prior tasks. Many of these approaches are either memory-hungry, brittle to task order, or impractical for models with billions of parameters. What distinguishes the new work is that it tackles the problem entirely within the cheap, low-rank adapter framework, requiring no access to old data and no freezing of the base model.

Technically, the method tracks the principal subspaces spanned by the low-rank updates in each layer and computes a similarity score between the subspace of an incoming task and those of earlier tasks. If the new task’s gradient updates are projected into directions that overlap heavily with prior subspaces, the rank of the adapter in that layer is expanded, giving the optimizer room to find solutions that spare old representations. Layers whose subspaces remain naturally disjoint need no expansion, so the parameter budget is spent only where it matters. The paper’s supplementary analyses reveal that the learned rank distributions differ markedly across layers and modules — for example, the variation in ranks among the value projection modules of encoder layers is larger than among query projections, while encoder layers as a whole show more consistent rank allocations than decoder layers. This suggests that different parts of a transformer genuinely serve distinct roles, and that a uniform rank silently wastes capacity in some layers while starving others.

To test the idea, the researchers ran experiments on T5 and Llama-2-7b language models, using GPU servers with DeepSpeed for efficient training. For T5 experiments they employed four NVIDIA A6000 GPUs with a learning rate of 1e-3 and a batch size of 32, while the larger Llama-2-7b runs used four NVIDIA A100 GPUs at a learning rate of 1e-4. The evaluation spanned standard natural language processing continual learning benchmarks built from 15 datasets, including the classic text classification benchmark of Zhang and colleagues, the GLUE and SuperGLUE suites, and the IMDB movie review corpus, with six different task sequence orders to control for ordering effects. The team also pushed into harder territory with challenging mathematical reasoning benchmarks, GSM8K and MATH, each split into sequential tasks and tested in both orders to probe how task sequencing shapes forgetting.

The results show that the adaptive-rank method matches or beats strong baselines on three fronts at once: it mitigates forgetting of earlier tasks, improves accuracy on the tasks being learned, and preserves strong generalization to unseen data — all in a memory-efficient manner. On the math benchmarks, where sequential training on GSM8K and MATH typically causes steep declines in earlier-task accuracy, the method with orthogonal projection consistently outperformed the O-LoRA baseline in final average testing accuracy, forgetting less on the first task while generalizing better on the second. Notably, the study observes that fully fine-tuning a T5-large model on MATH yields only about 3.0 percent accuracy, and about 4.2 percent on GSM8K, consistent with prior work — underscoring how difficult these mathematical tasks are and how much room remains for continual learning approaches that squeeze more capability out of small adapters.

Why should this matter beyond the benchmark tables? Continual learning is arguably the missing ingredient between today’s static AI assistants and the adaptive, lifelong learning systems that both researchers and companies envision. Every time a deployed model needs a new skill — a new language, a new domain, a new tool — retraining it from scratch or even fully fine-tuning it is prohibitively expensive. Adapter-based continual learning offers a path to modular, on-demand skill acquisition, but only if adding skills does not corrupt existing ones. By showing that the humble LoRA rank, tuned per layer and per task, is a powerful lever for balancing stability and plasticity, the Penn State work gives practitioners a new, inexpensive knob to turn. Because the method requires no stored examples of past tasks, it also sidesteps the privacy and storage concerns that plague replay-based approaches.

The findings also carry a broader scientific message: the internal geography of large models is far from uniform, and adaptation schemes that respect that heterogeneity can outperform uniform ones. The observation that encoder and decoder layers, and even the query and value modules within them, prefer different rank allocations suggests that future parameter-efficient methods may benefit from treating adapters as structured, layer-aware components rather than interchangeable plug-ins. The authors have released their code publicly, and all datasets used in the experiments are openly available, making the results straightforward for other groups to verify and extend. The work was partially supported by the National Science Foundation under award EFMA-2318101, with experiments designed and conducted by Fuli Qiao under the supervision of Mehrdad Mahdavi.

As large language models continue to spread through science, medicine, and industry, the question of how to teach them new things without breaking old ones is shifting from academic curiosity to engineering necessity. This study does not solve continual learning outright — no single method does — but it demonstrates that a careful, geometry-aware allocation of low-rank capacity can deliver competitive anti-forgetting performance at a fraction of the cost of full retraining. In a field where progress often comes from scaling up, it is a reminder that sometimes the most powerful improvements come from scaling the right things in the right places, one layer at a time.

Subject of Research: Adaptive layerwise LoRA ranks for continual learning in large language models

Article Title: Learning Without Forgetting for Continual Learning in LLMs Through Adaptive LoRA Ranks

Article References: Qiao, F., & Mahdavi, M. (2026). Learning Without Forgetting for Continual Learning in LLMs Through Adaptive LoRA Ranks. Machine Learning, 115(9), Article 212. https://doi.org/10.1007/s10994-026-07145-4

Image Credits: AI Generated

DOI: 10.1007/s10994-026-07145-4

Keywords: continual learning, large language models, LoRA, catastrophic forgetting, parameter-efficient fine-tuning, adaptive ranks, subspace orthogonality, stability-plasticity, GLUE, GSM8K, MATH, T5

At the core of the scPEFT framework is the integration of learnable, low-dimensional adapters into the architecture of existing scLLMs. This strategy is ingenious in its simplicity—by effectively freezing the backbone model and only updating the parameters associated with these new adapters, scPEFT can adapt the model to specific tasks using only a limited amount of custom data. This process allows researchers to harness the power of scLLMs without the extensive computational resources typically required for full model retraining.

Moreover, this framework effectively mitigates the issue of catastrophic forgetting, a common pitfall associated with conventional machine learning approaches where fine-tuning on new data leads to a degradation in the model’s performance on previously learned tasks. By focusing solely on the adapter parameters, scPEFT significantly reduces the overall parameter tuning by over 96%. This dramatic reduction not only streamlines the tuning process but also dramatically decreases memory requirements during training, making the technology much more accessible to researchers operating in resource-constrained environments.

The implications of scPEFT have been validated across a variety of datasets, revealing its superior performance compared to traditional zero-shot models and conventional fine-tuning techniques. The framework’s effectiveness is particularly pronounced in specialized applications, such as tasks relating to disease-specific analyses, cross-species studies, and the exploration of undercharacterized cell populations. These capabilities position scPEFT as a foundational advancement in the adaptation of scLLMs, enhancing their utility in real-world biological research scenarios.

One striking illustration of scPEFT’s power comes from its application in analyzing COVID-19-related genes. Through an attentional mechanism analysis, researchers were able to identify specific genes linked to particular states of cells, highlighting how scPEFT can lead to condition-specific interpretations that are vital for understanding the pathophysiology of diseases. This aspect of the framework showcases not only its scientific utility but also its potential to inform clinical strategies and therapeutic interventions.

In addition to its applications in infectious disease research, scPEFT has also unveiled unique blood cell subpopulations. The identification of these previously unrecognized cellular groups adds a new layer of understanding to hematological studies and could have significant implications for various fields, including oncology and immunology. The model’s capacity to discern subtleties within complex datasets reflects an evolution in single-cell analytics, ushering in a period where data-driven insights become more precise and actionable.

As researchers navigate the intricacies of cellular processes with this new tool, scPEFT holds the promise of significantly enhancing our understanding of cellular heterogeneity. By allowing for efficient adaptations of models tailored to specific biological questions, this framework could lead to breakthroughs in areas ranging from personalized medicine to developmental biology.

The introduction of scPEFT also signals a shift in the accessibility of advanced computational methods for the broader research community. Historically, the training of large models required substantial computational resources, which has acted as a barrier for many aspirant researchers. With the advantages of parameter-efficient fine-tuning, scPEFT democratizes access to powerful analytical capabilities, enabling a wider array of scientific inquiry without the need for extensive infrastructure.

Another key benefit of the scPEFT approach is its suitability for real-time application and rapid deployment in experimental settings. Given the fast-paced nature of many research fields, the ability to fine-tune models quickly and effectively can significantly expedite the discovery process. Researchers can expect faster turnaround times from hypothesis to results, thereby fostering a more dynamic and responsive scientific environment.

In sum, the emergence of scPEFT represents a remarkable advancement in the field of computational biology and artificial intelligence. This framework not only enhances the performance of scLLMs but also expands their applicability to a broader range of scientific questions. As the paradigm of single-cell research continues to evolve, tools like scPEFT will be pivotal in advancing our understanding of complex biological phenomena, ultimately leading to improved health outcomes and innovative therapeutic strategies.

The development of scPEFT is indicative of an exciting future in the integration of AI with biological research. As models grow increasingly sophisticated and adaptable, the potential for transformative insights into human health and disease will only increase. Researchers are poised at the forefront of this technological revolution, armed with tools that promise to unravel the mysteries of biology at an unprecedented scale.

As the scientific community embraces this new methodology, the impact of scPEFT will become evident across multiple domains of research. The synergy between machine learning and biology exemplified through this framework highlights the ongoing evolution of scientific discovery, showcasing the vital role that cutting-edge technology plays in expanding our knowledge and enhancing our capacity to tackle pressing global health issues.

Researchers are encouraged to explore the possibilities that scPEFT has to offer, familiarizing themselves with its mechanisms and implementing it to accelerate their investigations. With this framework, the future of single-cell analysis looks remarkably bright, illuminating paths toward discoveries that could reshape our understanding of biology as a whole.

Subject of Research: Single-cell large language models and parameter-efficient fine-tuning.

Article Title: Harnessing the power of single-cell large language models with parameter-efficient fine-tuning using scPEFT.

Article References:

He, F., Fei, R., Krull, J.E. et al. Harnessing the power of single-cell large language models with parameter-efficient fine-tuning using scPEFT.
Nat Mach Intell (2025). https://doi.org/10.1038/s42256-025-01170-z

Image Credits: AI Generated

DOI: https://doi.org/10.1038/s42256-025-01170-z

Keywords: scLLMs, scPEFT, parameter-efficient fine-tuning, single-cell analysis, computational biology, machine learning, COVID-19 research, gene identification, biological insights, accessibility in research.