To understand why this matters, it helps to unpack the core technique the authors build upon: knowledge distillation. First formalized by Geoffrey Hinton and Oriol Vinyals in 2015, distillation is the practice of transferring what a large, capable neural network knows into a smaller or differently structured network. In anomaly detection, the idea takes on a clever twist. A student network is trained to mimic a teacher network on images of normal products only. Because the student has never seen defects, it learns to reproduce only the features of normality. When a defective image is later presented, the teacher recognizes the full scene while the student, blind to the anomaly, produces a mismatched output. The magnitude of that mismatch becomes an anomaly map, highlighting precisely where the defect lies. The elegance of the method is that it requires no examples of failure at all, only a reliable supply of normal ones.

Where AnomalyMHKD departs from earlier work is in how the knowledge transfer is orchestrated. The framework integrates two complementary modes of distillation. The first, self-distillation, enhances intra-model feature alignment: different layers or heads within the same network teach one another, forcing the model to develop internally consistent representations of what a normal image looks like. This kind of internal harmonization has proven valuable in self-supervised representation learning, where the network generates its own training signal from the structure of the data rather than from human annotations. The second mode, cross-distillation, enables collaborative learning between heterogeneous architectures. Here the authors pair a Vision Transformer, or ViT, with a ResNet-style convolutional network, allowing each to distill its knowledge into the other.

The pairing of these two architectures is more than a technical flourish. Vision Transformers process an image by dividing it into patches and reasoning about the relationships between them through self-attention, giving them a global, holistic view of the scene. Convolutional networks like ResNet, by contrast, build understanding hierarchically from local patterns, excelling at capturing fine-grained textures such as scratches, dents or surface discolorations. When these two ways of seeing are forced into agreement through cross-distillation, the resulting feature representation inherits the strengths of both: the global context of the transformer and the local precision of the convolutional network. For patch segmentation-based anomaly detection, the task of localizing exactly which region of an image is defective, that hybrid representation is precisely what the doctor ordered.

The third ingredient is what the authors call a hybrid mode that toggles the teacher between online and offline learning. In offline distillation, a teacher is trained first and then frozen, its knowledge handed down in a one-way transfer. In online distillation, teacher and student learn simultaneously, adapting to one another throughout training. Each strategy has trade-offs: offline teachers are stable but can become stale, while online teachers stay current but can destabilize training. AnomalyMHKD dynamically switches between the two, and, crucially, employs a dynamic teacher-freezing approach that locks the teacher in place once its contributions plateau. According to the paper, this dynamic freezing is the primary driver of the approximately 30 percent reduction in training cost, since compute is no longer wasted updating a teacher that has already delivered most of what it knows.

The entire pipeline is self-supervised, meaning the model extracts its learning signal from the images themselves rather than from curated labels. This is the property that makes the approach attractive for real-world deployment. In manufacturing settings, collecting and annotating defective samples is expensive, slow and sometimes impossible, since rare defects may not appear until a process drifts months after deployment. A detector that learns normality from unlabeled images and flags deviations can be retrained cheaply whenever a product line changes, and the authors note that the method achieves its strong results with less dependence on labeled data than supervised alternatives demand.

The benchmarks on which the framework was evaluated are the de facto proving grounds for industrial anomaly detection. MVTech AD, introduced by researchers at the Technical University of Munich, contains fifteen categories of objects and textures, from screws and metal nuts to tiles and leather, each photographed with a variety of subtle defects. VisA is a more recent and arguably harder dataset spanning twelve categories of complex scenes, including printed circuit boards and capsules, where anomalies can be small, low-contrast or structurally ambiguous. Scoring 96.6 percent and 93.6 percent AUROC respectively places AnomalyMHKD among the leading published methods, and the authors report that it compares favorably with recent state-of-the-art systems such as EfficientAD, a detector engineered for millisecond-level inference that was presented at the IEEE/CVF Winter Conference on Applications of Computer Vision in 2024.

The study situates itself within a rapidly evolving lineage. Reverse distillation methods, such as the one proposed by Deng and Li at CVPR 2022, inverted the traditional teacher-student arrangement to improve one-class embeddings, and follow-up work like SK-RD4AD in 2025 added skip connections for robustness. FastFlow applied two-dimensional normalizing flows to model the distribution of normal features, while PatchCore pursued nearest-neighbor memory banks toward what its authors called total recall. On the distillation side, multi-mode online knowledge distillation for self-supervised visual representation learning appeared at CVPR 2023, and CrossKD introduced cross-head distillation for object detection in 2024. AnomalyMHKD synthesizes these threads, combining multi-mode distillation with heterogeneous cross-architecture transfer and dynamic teacher management, and directs the resulting machinery at the specific problem of unsupervised anomaly localization.

The practical implications extend well beyond factory floors. The same self-supervised machinery that spots a scratched circuit board can, in principle, be adapted to medical imaging, where labeled pathology is likewise scarce, or to surveillance, infrastructure inspection and any domain where the abnormal is rare by definition. The authors state that the datasets analyzed in the study are publicly available from their respective sources, which lowers the barrier for other groups to reproduce and extend the results. The work also received no specific funding, and the authors declare no conflict of interest, with all four contributors listed as having contributed equally to the manuscript.

There remain, of course, the usual caveats that accompany any new benchmark result. The reported figures come from the authors’ own experiments, and independent replication on additional datasets and deployment conditions will be needed to confirm the promised 30 percent training cost savings in production environments. The method’s reliance on patch segmentation means its localization quality, not just its image-level detection scores, will matter to adopters who need to know not only that a part is defective but where. Still, the conceptual contribution is clear and timely: by letting a transformer and a convolutional network teach each other, by letting a model’s layers align themselves, and by freezing teachers the moment they stop being useful, AnomalyMHKD demonstrates that the path to reliable anomaly detection may run not through bigger labeled datasets, but through smarter ways of transferring the knowledge machines already possess. As factories, hospitals and cities generate ever more visual data with ever fewer annotations, that lesson is likely to echo far beyond the inspection line.

Subject of Research: Self-supervised anomaly detection using multi-mode hybrid knowledge distillation between Vision Transformer and ResNet models

Article Title: AnomalyMHKD: A multi-mode hybrid knowledge distillation approach for self-supervised detection of anomalous objects

Article References: Bhuvana, J., Mirnalinee, T. T., Mohan, H., & M, O. (2026). AnomalyMHKD: A multi-mode hybrid knowledge distillation approach for self-supervised detection of anomalous objects. Multimedia Tools and Applications, 85(9), Article 745. https://doi.org/10.1007/s11042-026-21844-z

Image Credits: AI Generated

DOI: 10.1007/s11042-026-21844-z

Keywords: anomaly detection, knowledge distillation, self-supervised learning, Vision Transformer, ResNet, MVTech AD, VisA, industrial inspection, computer vision, EfficientAD, patch segmentation, teacher-student networks

A team led by Linh Do Nhat, Dzi Lam Tran Tuan and Vinh Truong Hoang has now published EmbedKD, an open-source Python toolkit designed specifically for distilling metric embeddings, in the journal SoftwareX. The software, released under the MIT license and available on GitHub, deliberately contributes no new distillation objective. Instead, its authors argue, the real gap in the field is the lifecycle around the methods: the diagnostics that reveal whether a teacher–student pair is compatible before training, the retrieval protocols that measure whether distillation actually helped, and the deployment checks that verify the model leaving the pipeline still behaves like the one that was trained. Every component, they note, could be rebuilt inside existing frameworks with moderate effort; what EmbedKD provides is that workflow, verified stage against stage.

The distinction matters because existing tools were built for a different problem. Frameworks such as torchdistill, mdistiller and KD-Lib offer config-driven pipelines, official implementations and combinations with pruning and quantization, but none ships a retrieval protocol with mean average precision and recall metrics, a gallery-query evaluation, pre- and post-training diagnostic reports, or a deployment benchmark with a parity check. Retrieval-oriented distillation methods themselves are well developed, from multi-view distillation for re-identification to pairwise-difference relational distillation, but each is released as code accompanying a single paper’s experiments. EmbedKD instead wraps the process in a six-command lifecycle: diagnose, fit, eval, extract, deploy and reproduce, each of which can be run from the command line or through a shared Python API that guarantees the two paths produce identical numbers.

Reproducibility is treated as a hard requirement rather than an aspiration. Every execution records its full state, including configuration, random seed, software versions, code commit and hardware, and all published results trace back to these logs. The reproduction command grades its output against tolerance-checked expected values and exits with a distinct error code on any metric drift, allowing continuous-integration systems to gate on it. In one demonstration, retraining the reference experiment returned results identical to the original run in every digit rather than merely within tolerance. The released version is archived on Zenodo with checksums for every checkpoint, and the reference environment, down to driver and package versions, ships with the repository.

Architecturally, the toolkit rests on three decisions. A registry system lets users add distillation objectives, task losses and dataset adapters as first-class citizens of the configuration file without touching the core. A single declarative configuration schema merges defaults, YAML files and command-line overrides into one validated, fail-fast configuration that is dumped verbatim into the run fingerprint. And an explicit backbone policy distinguishes five validated teacher and student architectures tested in continuous integration, nineteen models that pass smoke tests, and any other timm backbone, which runs only under an explicit experimental flag with a warning rather than a silent claim of support.

The toolkit also surfaces numerical subtleties that are easy to get wrong. Because the supported objectives operate on L2-normalized embeddings, mean squared error and cosine distance are the same function up to a constant, but the reduction over the embedding dimension means that at a dimensionality of 512, the same nominal weight makes the MSE distillation term 256 times weaker than the cosine term. Experiments on the CUB-200-2011 bird dataset confirmed the algebra: at the nominal weight the MSE objective changed nothing, while at the scale-matched weight it matched cosine distillation almost exactly. The toolkit prints the effective weight of each objective when a run starts, computes all losses in fp32 outside the autocast region because geometric losses underflow in fp16 and silently freeze training, and skips and counts non-finite loss components rather than poisoning a training step.

Perhaps the most striking findings come from the diagnostics themselves, which the authors report with unusual candour. In a controlled series where only teacher quality varied, a teacher weaker than the standalone student cost the student 0.023 mAP while its representation became more aligned with the teacher, a failure mode the team calls aligned-but-worse. The sign of the distillation effect changed near a teacher–student gap of 0.02 to 0.03 and saturated after roughly 0.05. But across different student architectures the gap failed to predict the magnitude of the gain, and a stronger ConvNeXt-Tiny teacher yielded smaller gains than ResNet50 for all three students tested. Most sobering, the pre-distillation risk label, intended to warn of harmful pairs, read LOW on every rung of the controlled series including the harmful one, and its underlying similarity score shifted enough between probe splits to cross its own threshold.

The cross-domain results were equally cautionary. When students trained on one dataset were evaluated on another without fine-tuning, distillation hurt in both directions tested: a CUB-trained ResNet18 lost 0.0074 mAP on Cars196, and a Cars-trained MobileNetV3 lost 0.0320 on CUB-200, both far outside the seed spread, even though both pairs had gained in domain. The distillation term, the authors suggest, pulls the student toward a teacher fitted to the source domain, and the pull that helps within a domain costs generality when the domain changes. For the deployment scenario, however, the news was better: a distilled MobileNetV3 kept 72.4 percent of its teacher’s Recall@1 while being 5.7 times smaller and eight times faster on CPU, with ONNX export verified by a mandatory parity check that recomputes retrieval metrics through the exported graph.

The authors are explicit about the limits of their evidence base: four demonstrations, one controlled series and one six-pair grid, all on a single machine, with several comparisons resting on single runs. The toolkit targets single-GPU image workloads, and the shipped training schedule trains past the best checkpoint for every backbone measured, requiring validation-based checkpoint selection. Yet the broader message is one the field is increasingly willing to hear. Whether a given embedding pair is worth distilling is today settled by trial and error, one configuration at a time; EmbedKD makes it cheap to settle by measurement instead, turning questions usually answered by folklore into routine, reproducible experiments, and, just as importantly, documenting where its own instruments cannot see.

Subject of Research: A reproducibility-first open-source toolkit for knowledge distillation of metric embeddings in image retrieval.

Article Title: EmbedKD: a reproducibility-first toolkit for knowledge distillation of metric embeddings

Article References: Do Nhat, L., Lam Tran Tuan, D., & Truong Hoang, V. (2026). EmbedKD: a reproducibility-first toolkit for knowledge distillation of metric embeddings. SoftwareX, 36, Article 103029. https://doi.org/10.1016/j.softx.2026.103029

Image Credits: AI Generated

DOI: 10.1016/j.softx.2026.103029

Keywords: knowledge distillation, metric embeddings, image retrieval, reproducibility, model compression, open-source software, deep metric learning, teacher-student networks, ONNX deployment, negative transfer, representation similarity, fine-grained recognition