When AI Forgets, Rare Diseases Pay the Price: New Audit Exposes Hidden Risks in Medical Model Maintenance

When a hospital asks an artificial intelligence vendor to delete a patient’s data, the expectation is simple: the model should forget that record and nothing else should change. In practice, according to a new study published in the International Journal of Data Science and Analytics, forgetting is anything but surgical. The research, conducted by Ismail Hakki Kinalioglu of Selcuk University, introduces a systematic audit showing that the process of removing training records from medical AI models — known as machine unlearning — can quietly devastate the model’s ability to recognize rare diseases, even when overall performance appears intact. The finding strikes at a growing tension in healthcare AI: regulators and privacy laws increasingly demand the right to erasure, while clinical safety demands that models remain reliable for every patient, including those with uncommon conditions.

The core problem the study addresses is what the author calls rare-disease spillover. When an unlearning procedure is applied to a medical image classifier, the update does not merely erase the targeted records. Gradient-based deletion methods reshape the model’s internal decision boundaries, and those boundaries are often far more fragile for minority classes — the rare diagnoses that appear in only a handful of training images — than for common ones. A model can lose a substantial fraction of its rare-class accuracy while its aggregate F1 score, the metric most dashboards report, barely moves. The study formalizes this discrepancy with a statistic called the Rare-Disease Spillover Index, or RDSI, which measures the differential change in utility experienced by rare classes relative to the model’s overall performance after an unlearning update. Crucially, the author frames the RDSI as a reporting convention rather than an independent source of information, a deliberate design choice meant to prevent the statistic from being treated as a standalone certification of safety.

The experimental evidence is stark. Working with HAM10000, a widely used openly available dataset of dermatoscopic images of pigmented skin lesions, the study used a fixed lesion-disjoint partition — meaning images from the same patient’s lesion never straddle the training and evaluation sets, a detail that matters enormously in dermatology where multiple images of one lesion can otherwise inflate apparent performance. On this partition, an aggressive gradient-ascent unlearning method known as NegGrad produced an F1-based RDSI of 0.373, with a paired lesion-bootstrap confidence interval of [0.291, 0.449]. In plain terms, the rare classes suffered a large and statistically robust drop in utility that would be invisible in aggregate metrics. By contrast, ordinary retraining on the retained data alone — the naive baseline of simply rebuilding the model without the deleted records — yielded an RDSI of just 0.052, with an interval spanning [-0.014, 0.114] that includes zero. The gap between these two numbers carries the study’s central message: how successfully a model forgets and how much collateral damage it inflicts are two separate questions that demand separate assessment.

This distinction has direct regulatory weight. Privacy frameworks such as the GDPR and emerging AI governance rules treat data deletion as a compliance obligation, and unlearning methods are attractive precisely because retraining a large clinical model from scratch for every deletion request is expensive and slow. But the audit shows that a fast unlearning update can satisfy the letter of a deletion request while quietly degrading the model’s clinical value for the patients who need it most. Rare diagnoses — unusual skin cancers, uncommon radiographic findings — are exactly the cases where clinical AI assistance is most valuable and where the training data is thinnest. A deletion procedure that erodes rare-class performance is therefore not a neutral maintenance operation; it is a potential patient-safety event that no current compliance checklist would catch.

The study does not stop at diagnosis. It also evaluates a mitigation strategy that combines two ideas: rare replay, in which a small, carefully chosen set of retained rare-class examples is revisited during the unlearning update to anchor the model’s rare-class behavior, and prototype anchoring, a technique borrowed from few-shot learning research in which class prototypes — averaged feature representations — serve as reference points that stabilize the encoder’s representation of rare categories. The illustrated replay-plus-anchor update achieved a test macro-F1 of 0.675 with a standard deviation of 0.020 across seeds, and a rare macro-F1 of 0.624 with a standard deviation of 0.038. For comparison, fresh retain-only retraining with the same rare-class-weighted loss function reached only a rare macro-F1 of 0.460 with a standard deviation of 0.029. The mitigation update also achieved a forget-set accuracy of 0.975 with a standard deviation of 0.014, versus 0.750 with a standard deviation of 0.020 for matched retraining — a number that, counterintuitively, signals residual predictive behavior on the forgotten images rather than successful deletion.

That last point deserves unpacking, because it reveals a subtle trap in unlearning evaluation. High accuracy on a forget set sounds like good news if you are measuring utility, but in the unlearning context it is a red flag: a model that still classifies deleted images accurately has not truly forgotten them. The study complements its utility comparisons with uniform probability distances, confidence diagnostics, and a retain-forget membership inference analysis — an adversarial test that asks whether an attacker can distinguish images the model trained on from images it did not. Successful unlearning should collapse that separability, making deleted records indistinguishable from never-seen ones. By pairing utility metrics with deletion-fidelity metrics, the audit framework forces both questions to be answered simultaneously, exposing methods that look safe on one axis while failing on the other.

Perhaps the most sobering result concerns the limits of the mitigation itself. In paired centroid-source controls, a retain-only encoder achieved a rare macro-F1 of 0.592 with a standard deviation of 0.033, compared with 0.608 with a standard deviation of 0.040 for the full-data encoder and 0.572 with a standard deviation of 0.022 for replay alone. In other words, simply matching the rare replay data does not reproduce the operating point achieved by the full mitigation pipeline, and swapping the anchor reference produced only a modest, uncertain incremental gain in utility. The author is candid about this: the mitigation is illustrative, not a finished prescription. The deeper contribution is the audit structure itself — raw class utilities, paired uncertainty quantification through lesion-level bootstrapping, and explicit retraining controls that let practitioners distinguish genuine rare-class preservation from artifacts of the evaluation setup.

The study also looked beyond dermatology, testing external chest-radiograph effects using openly available data from ChestX-ray14 and a credentialed subset drawn from MIMIC-CXR-JPG. There, the spillover effects were modest and heterogeneous, suggesting that the severity of rare-disease collateral damage depends on the dataset, the class structure, and the unlearning method. That heterogeneity is itself an argument for the audit: there is no universal safety margin that a hospital can assume when a model update deletes data. Each deletion event, on each deployed model, needs its own paired assessment of what was lost and whether the loss falls disproportionately on rare conditions.

What makes the work unusually verifiable is its commitment to reproducibility. The full pipeline — partitioning, training, unlearning, mitigation, auditing, and every reported calculation — is archived on Zenodo under an MIT license, complete with fixed split manifests, per-seed results, bootstrap distributions, and compact per-image predictions. The reported numbers can be recomputed without images, GPUs, or model checkpoints, and the credentialed chest-radiograph components are handled through identifiers and reconstruction scripts rather than redistributed data. In a field where unlearning claims are often difficult to check, this level of transparency sets a practical standard.

The broader implication is that healthcare AI needs a maintenance discipline analogous to pharmacovigilance. Models are not static artifacts; they are edited throughout their deployment lives as privacy requests arrive, datasets are corrected, and regulations evolve. The study’s framework treats each such edit as a potential safety event requiring a structured audit: measure raw per-class utilities before and after, quantify uncertainty at the level of correlated patient clusters, run explicit retraining controls, test deletion fidelity against membership inference, and report rare-class spillover alongside aggregate performance. As medical AI systems move from research demos into clinical infrastructure, the question is no longer only whether a model can forget on command, but whether the rest of the model — and the rare patients it serves — survives the forgetting intact. This audit provides the first systematic vocabulary for asking that question, and a reproducible toolkit for answering it.

Subject of Research: Machine unlearning safety in healthcare AI, focusing on rare-disease performance degradation and mitigation

Article Title: Rare-disease spillover after medical AI unlearning: a safety audit and mitigation framework for healthcare model maintenance

Article References: Rare-disease spillover after medical AI unlearning: a safety audit and mitigation framework for healthcare model maintenance. (n.d.). https://doi.org/10.1007/s41060-026-01329-w

Image Credits: AI Generated

DOI: 10.1007/s41060-026-01329-w

Keywords: machine unlearning, healthcare AI, rare diseases, dermoscopy, HAM10000, membership inference, model maintenance, data deletion, medical imaging, safety audit, class imbalance, deep learning