New Training Trick Lets Federated AI Forget a Client Instantly and Perform Better

Federated learning has become one of the most widely deployed paradigms for collaborative machine learning, allowing hospitals, banks, and phone manufacturers to train a shared model without ever moving raw data off their premises. Clients exchange only model updates such as gradients or parameters with a central server, which aggregates them into a global model. Yet this privacy-preserving design creates a stubborn problem: once a client’s updates have been woven into the shared model over many communication rounds, that client’s influence is baked in. Removing it, whether because of a legal erasure request or a discovered poisoning attack, has traditionally meant retraining everything from scratch. A new study published in the journal Machine Learning by Ze Yu Zhang, Nhung Bui, Arun Verma, Bolin Ding, and Bryan Kian Hsiang Low proposes an elegant escape from this dilemma, introducing two algorithms called Bi-Models Training and Multi-Models Training that promise exact forgetting by design, without sacrificing the model’s accuracy afterward.

The stakes are higher than they might first appear. Privacy regulations such as the European Union’s General Data Protection Regulation grant individuals a Right to be Forgotten, and similar obligations exist under the California Consumer Privacy Act. When a participant in a federated system demands erasure, the operator cannot simply delete the client’s local files; the fingerprints of that data persist in the aggregated global model. Membership inference attacks and gradient reconstruction attacks have demonstrated that such residual traces can leak sensitive information. In security-sensitive settings the problem is even more acute: a single malicious or poisoned client can corrupt a jointly trained fraud detection model, and eliminating its influence becomes essential to restoring the model’s reliability. In both scenarios, the removal must be complete, not approximate.

The gold-standard solution, retraining from scratch with the remaining clients, does achieve exact unlearning, because a model trained only on the surviving data is provably independent of the departed client’s dataset. But the cost is prohibitive. Federated training typically runs for hundreds to a thousand communication rounds, and restarting the process introduces significant deployment delays. Worse, the random initialization that begins a fresh training run can lead to severe performance degradation, particularly when the remaining client population is small or heterogeneous. For time-sensitive applications, from fraud detection to medical imaging, waiting days for a retrained model is simply not an option. This gap between the guarantees of retraining and the practical need for immediacy is precisely where the new work positions itself.

The first of the two proposed algorithms, Bi-Models Training, takes a surprisingly simple approach. During the normal federated training process, each client maintains an isolated local copy of the model that is trained only on its own data, alongside its participation in the shared global training. These local copies are never mixed with anyone else’s data, so their influence sets contain exactly one client each. When a client later requests to be forgotten, the server simply discards every stored state influenced by that client and uses the remaining clients’ isolated local models to provide a strong initialization for the post-unlearning global model. Instead of starting from random weights, the recovered model starts from weights that already encode useful knowledge learned by the surviving participants, dramatically shortening the recovery phase and improving final accuracy.

Bi-Models Training, however, treats each client in isolation, which means it cannot capture how the influence of multiple clients is jointly entangled in the global model. This is where Multi-Models Training, the paper’s centerpiece, departs from convention. Rather than training a single global model, the server orchestrates a carefully constructed set of sub-federated models, each trained on a disjoint subset of clients. Because the subsets do not overlap, no client’s influence spreads across multiple sub-models, and the influence structure of the entire system can be represented as a tree whose leaves are the individual clients. When a client must be removed, the algorithm identifies the most influential models that remain unaffected by that client and aggregates them to reinitialize the global model, guaranteeing that the departed client’s data plays no role in the result.

One of the most striking theoretical contributions of the paper is the connection it draws between the design of these influence trees and a classic result from information theory. The authors prove that the expected number of models that must be aggregated after an unlearning event, a quantity they call the influence-dependent storage cost, is minimized when the tree of sub-models is constructed according to Huffman coding, the same greedy algorithm used to build optimal prefix-free compression codes. In other words, the optimal way to partition clients into disjoint training groups mirrors the optimal way to assign short binary codewords to frequent symbols. This unexpected bridge between compression theory and federated unlearning gives the method a principled foundation rather than a heuristic one, and it allows the system to exploit known probabilities about which clients are likely to request erasure.

The guarantees of exactness are established formally. The authors model training as a composition of deterministic primitives, including initialization, copying, optimizer resets, local training on a dataset, and server aggregation. They define the influence set of any model state as the set of clients whose data could have affected it, and they prove that a model state’s value depends on the training data only through this influence set. The unlearning procedure then discards every stored state whose influence set contains the departing client, aggregates the retained states, and continues training only on the remaining clients’ data. Because the departing client is never reintroduced into any influence set, the final deployed model is provably invariant to any replacement of that client’s dataset, which is precisely the formal definition of exact unlearning. Notably, the framework is compatible with arbitrary federated learning protocols, so it does not require modifying the underlying aggregation rules.

Memory efficiency is another decisive advantage. An earlier exact approach, Exact-Fun, stores auxiliary models for every communication round, and with typical federated training spanning one hundred to one thousand rounds this becomes extremely memory-inefficient, and in the authors’ experiments with Llama-3.2 language models, outright infeasible. By contrast, the storage cost of both Bi-Models Training and Multi-Models Training depends only on the number of clients, making them well suited to cross-silo deployments involving ten to one hundred participants, such as consortia of hospitals or financial institutions. The authors even arranged their evaluation order to disadvantage their own method, ensuring that the global model could not simply be replaced by a single sub-model and had to be reconstructed through aggregation, a choice that strengthens the credibility of their reported gains.

The empirical results span both vision and language tasks. On image classification benchmarks including CIFAR-10, Fashion-MNIST, and datasets evaluated with VGG-style convolutional networks, as well as language tasks including language identification and multilingual sentiment classification drawn from public Hugging Face datasets and experiments fine-tuning Llama-3.2, Bi-Models Training consistently outperformed existing exact unlearning baselines. Multi-Models Training went further, achieving more than a twofold improvement in post-unlearning performance while maintaining the exact unlearning guarantee. The comparison baselines included approaches such as FedEraser, Exact-Fun, and federated unlearning via weight negation, alongside the retraining-from-scratch reference point. The consistent pattern across tasks suggests that the benefit stems from the initialization strategy itself rather than from any quirk of a particular dataset or architecture.

The implications reach well beyond the immediate technical contribution. As federated learning spreads through healthcare, open banking, mobile keyboards, and recommendation systems, the ability to honor erasure requests instantly, and to surgically excise poisoned participants from security-critical models, is becoming a baseline requirement rather than a luxury. The work also highlights a subtle vulnerability: unlearning systems themselves can be targets, with recent research demonstrating data reconstruction attacks specifically aimed at federated unlearning procedures. By making exactness a structural property of the training process rather than a post-hoc repair, the new algorithms close a gap that approximate methods leave open. The research, supported by Singapore’s National Research Foundation and the Singapore-MIT Alliance for Research and Technology, suggests a future in which the right to be forgotten and the demand for high-performing collaborative models are no longer in tension, but are satisfied simultaneously by design.

Subject of Research: Exact federated unlearning algorithms that remove a client's influence from collaboratively trained models while improving post-unlearning performance

Article Title: MMT: Achieving Exact Federated Unlearning with Improved Post-unlearning Performance

Article References: MMT: Achieving Exact Federated Unlearning with Improved Post-unlearning Performance. (n.d.). https://doi.org/10.1007/s10994-026-07117-8

Image Credits: AI Generated

DOI: 10.1007/s10994-026-07117-8

Keywords: federated learning, machine unlearning, exact unlearning, data privacy, GDPR, Huffman coding, model poisoning, Llama-3.2, influence trees, Machine Learning journal, federated protocols, post-unlearning performance