The research, published in Multimedia Tools and Applications by Melisa Alara Ozuberk and Ilkay Cinar, delivers one of the first head-to-head evaluations of three leading real-time open-vocabulary detectors: YOLO-World, its successor YOLO-World v2, and YOLOE. Rather than benchmarking accuracy alone, the authors designed their experiments around three scenarios that mirror how these systems are actually deployed: zero-shot inference on entirely new datasets, sensitivity to variations in the textual prompts that steer detection, and fine-tuning on domain-specific data followed by tests of whether open-vocabulary generalization survives. The evaluation spans three datasets with deliberately different characteristics—the classic VOC2012 segmentation subset, the HomeObjects-3K indoor detection dataset, and the demanding KITTI autonomous driving benchmark.

The zero-shot results reveal a striking dependence on domain. The highest performance was achieved on HomeObjects-3K, where YOLO-World v2 reached 0.443 mAP@0.5:0.95, a metric that rewards both accurate localization and correct classification across a range of overlap thresholds. KITTI, by contrast, produced the weakest results across all three models, a consequence of domain shift: the driving imagery, with its unusual viewpoints, small distant objects, and harsh lighting conditions, differs substantially from the data distributions these models encountered during pre-training. On the VOC dataset, YOLOE claimed the highest zero-shot accuracy at 0.310 mAP@0.5:0.95, outperforming both YOLO-World variants on that benchmark, although this advantage came with a caveat that emerged clearly in the timing analysis.

That caveat is speed. YOLO-World v2 proved to offer the best overall balance between accuracy and throughput, sustaining between 20 and 30 frames per second—comfortably real-time for many applications. YOLOE, despite its stronger zero-shot accuracy on VOC, exhibited lower inference speed in some configurations, a trade-off that could prove decisive in latency-sensitive settings such as autonomous navigation or live video analytics. The YOLO-World family also benefited from an embedding cache mechanism, which pre-computes text embeddings for the prompt vocabulary and reuses them across frames. Latency analysis with increasing prompt counts showed that this design keeps inference efficient even as the number of textual categories grows, an architectural advantage that becomes more valuable the richer the vocabulary deployed in production.

Perhaps the most practically important finding concerns prompt robustness—the question of how much detection quality degrades when the words fed to the model change. The authors constructed four categories of prompts for each dataset: base prompts, attribute prompts that add descriptive modifiers, longer descriptive prompts, and noisy prompts containing degraded or perturbed language. The results showed measurable performance drops under noisy prompt conditions for all models, confirming that open-vocabulary detectors are not immune to the fragility of language interfaces that has been documented across the broader vision-language literature. However, YOLO-World v2 maintained better stability across these variations than its competitors, suggesting that its training recipe or text-encoding pathway confers a degree of resilience that practitioners should weigh when deploying systems in the hands of non-expert users who cannot be relied upon to craft optimal prompts.

Fine-tuning delivered the expected gains but also exposed an uncomfortable truth about what adaptation costs. After fine-tuning on each dataset, mAP scores improved for all three models, demonstrating that standard transfer learning techniques remain effective when open-vocabulary detectors are specialized to a target domain. But the fine-tuned models showed zero performance on some unseen categories—classes that were never part of the fine-tuning data. This is precisely the failure mode that open-vocabulary detection is supposed to prevent, and its appearance after adaptation indicates that the boundary between open and closed vocabulary is thinner than the field often assumes.

The divergence between the model families was especially pronounced here. The YOLO-World family retained some of its open-vocabulary generalization ability after fine-tuning, continuing to respond to textual prompts for categories outside the training set. YOLOE, under the fine-tuning protocol adopted in the study, exhibited closed-set-like behavior: its predictions remained insensitive to the evaluated prompt variations, effectively behaving as if the text interface had been switched off and the model had reverted to a conventional fixed-category detector. For teams choosing between these architectures, the implication is significant—fine-tuning YOLOE may buy accuracy on known classes at the price of the very flexibility that motivated choosing an open-vocabulary model in the first place.

The study’s methodology reflects a growing recognition that evaluation practices in this field have been too narrow. The authors note that existing research has focused mainly on accuracy metrics while prompt robustness, unseen class generalization, and computational costs are rarely assessed together. By combining confusion-matrix-based analysis, standard detection metrics such as mAP at multiple intersection-over-union thresholds, and latency profiling under varying prompt counts, the work offers a template for more honest benchmarking. The datasets themselves are all publicly available—VOC2012 from the PASCAL repository, KITTI from the KITTI Vision Benchmark Suite, and HomeObjects-3K through its original repository—making the evaluation pipeline reproducible by other groups.

The broader context makes these findings timely. Open-vocabulary detection builds on a lineage that runs from the original YOLO real-time detector through open-set recognition and open-world detection to caption-supervised methods and the CLIP-style vision-language models that supply the text-image alignment these detectors depend on. YOLO-World, introduced in 2024, brought this capability to real-time speeds, and YOLOE pushed the concept further with its “see anything” design. Applications documented in the literature now span automatic image annotation, number plate recognition, wildlife monitoring, medical imaging, underwater fish counting, robotic navigation, and anomaly detection in surveillance—domains where the ability to name new categories without retraining is transformative.

For practitioners, the study’s bottom line is that model selection should be a multi-dimensional decision. Accuracy, inference speed, prompt robustness, and unseen class generalization form a set of trade-offs that no single model dominates. YOLO-World v2 emerges as the most balanced option, combining competitive accuracy, real-time throughput, an efficient embedding cache, and the strongest prompt stability. YOLOE offers the best zero-shot accuracy in some settings but pays in speed and, critically, appears to surrender its open-vocabulary character when fine-tuned. As these systems move from research demos into safety-relevant deployments—self-driving perception, medical triage, industrial inspection—the lesson of this comparative study is that the questions worth asking about a detector extend well beyond its leaderboard score, reaching into how it behaves when the words change, the domain shifts, and the training data runs out.

Subject of Research: Comparative evaluation of prompt robustness, fine-tuning, and generalization in open-vocabulary object detection models

Article Title: Prompt robustness, fine-tuning, and Generalization in open-vocabulary object detection: a comparative study of YOLO-World, YOLO-World v2 and YOLOE

Article References: Ozuberk, M. A., & Cinar, I. (2026). Prompt robustness, fine-tuning, and Generalization in open-vocabulary object detection: a comparative study of YOLO-World, YOLO-World v2 and YOLOE. Multimedia Tools and Applications, 85(10), Article 767. https://doi.org/10.1007/s11042-026-21928-w

Image Credits: AI Generated

DOI: 10.1007/s11042-026-21928-w

Keywords: open-vocabulary detection, YOLO-World, YOLOE, zero-shot object detection, prompt sensitivity, fine-tuning, computer vision, mAP, inference latency, domain shift, vision-language models, object recognition

The foundation of the project is Mindat, the sprawling online mineral database that has quietly become the reference shelf of the entire mineralogy community. As of February 2025, Mindat cataloged 6,114 mineral species approved by the International Mineralogical Association, accompanied by nearly 1.41 million high-quality photographs and more than 140 distinct mineral attributes. That ocean of images has always been searchable by name or locality, but the relationships between the pictures and the text describing them remained largely unexploited. The researchers, led by Quanli Fu and Xiang Que of Fujian Agriculture and Forestry University, together with Xiaogang Ma of the University of Idaho and colleagues, saw in those unexplored connections an invitation: if every photograph is tied to a structured description of hardness, luster, cleavage and chemistry, could a machine learn to run the relationship in reverse?

Their answer builds on the OpenMindat project, an effort funded by the U.S. National Science Foundation to expose the Mindat database through a machine-readable interface that complies with the FAIR principles of findability, accessibility, interoperability and reusability. Earlier work in the OpenMindat ecosystem produced R and Python packages, a mobile application, and visualization frameworks for exploring associations among minerals, elements and localities. What had never been attempted, the authors note, was the integration of Mindat’s textual attribute data with its images to generate entirely new mineral pictures. Text-to-image synthesis, the technology behind tools like DALL-E and Midjourney, had already proven itself in medical imaging, remote sensing and design education; nobody had pointed it at the mineral kingdom.

The first task was data engineering, and it was substantial. The team pulled 10 to 20 images for each of the 6,114 IMA-approved mineral species, collecting every available photograph for species with fewer than ten, and ended up with a dataset of 43,745 images, which they have released publicly on Hugging Face. Each image was paired with a standardized textual description assembled from ten attributes drawn from four categories: chemical composition, physical characteristics, crystal system and optical behavior. The template reads like a telegraphic mineralogist’s shorthand: a mineral’s transparency, constituent elements, Mohs hardness range, luster type, streak color, crystal system, cleavage, fracture and optical type are stitched into a single sentence. Attributes with missing values are simply omitted, and the image-to-text pairing was verified by matching image folder names against mineral names returned by the Mindat API.

With the dataset in hand, the researchers fine-tuned Stable Diffusion, a prominent text-to-image model built on the Latent Diffusion framework. Stable Diffusion’s architecture has three moving parts that matter here: a variational autoencoder that compresses images into a compact latent space and reconstructs them, a U-Net that learns to strip Gaussian noise away step by step, and a CLIP text encoder that converts prompts into vector representations injected into the U-Net through cross-attention layers. During training, the model learns to predict the noise corrupting a latent image, conditioned on the text prompt, so that at generation time it can start from pure noise and denoise its way toward an image that matches the description. Fine-tuning concentrated on the U-Net, leaving the pre-trained VAE and text encoder frozen, since those components already carry strong general-purpose capabilities.

The team compared two fine-tuning strategies. Full fine-tuning updates every parameter of the U-Net, a comprehensive adjustment that in this study involved 859.52 million trainable parameters. Low-Rank Adaptation, or LoRA, takes a leaner path: instead of rewriting the weight matrices outright, it freezes them and learns small low-rank residual matrices whose product approximates the needed update, cutting the trainable parameter count to just 0.79 million. LoRA’s efficiency has made it the darling of the open-source AI community, but the mineral experiment delivered a clear verdict. Evaluated with the Fréchet Inception Distance, the Kernel Inception Distance and CLIPScore, the fully fine-tuned model outperformed both the LoRA variant and the untouched baseline, achieving an FID of 48.83 and a KID of 0.02, indicating generated images whose statistical distribution sits closest to that of real mineral photography.

The qualitative differences were just as telling. The baseline Stable Diffusion model, asked for a mineral, tended to produce geometric patterns with excessive specular reflections and a synthetic, manufactured sheen, while the Kandinsky model drifted toward an artistic style with morphologically concentrated structures. The fully fine-tuned model, by contrast, produced images with irregular granular textures, heterogeneous color distributions and distinct crystalline structures that echo real specimens. LoRA outputs improved on the baseline but remained unstable, sometimes over-smoothed and incompletely conditioned on the target attributes. The authors caution that CLIPScore, being trained on general-domain corpora, may understate the gains on specialized mineralogical vocabulary, a known limitation of automatic metrics when domain-specific terminology enters the picture.

One of the study’s most practical findings concerns prompt detail. When the researchers varied how many attributes appeared in a prompt, from a terse three-attribute description to a comprehensive one matching the training template, generation quality climbed steadily with specificity. Sparse prompts leave the model under-constrained, allowing attribute combinations that could correspond to many different minerals, while richer descriptions tighten the visual target. The team validated the approach against reality by querying Mindat’s own advanced search portal for images of chrysoberyl matching a given attribute combination and finding the AI-generated counterparts strikingly similar. They then pushed further, feeding the model attribute combinations belonging to no known IMA-approved species, generating portraits of hypothetical minerals that may exist in nature but remain unclassified, or may only form transiently under extreme temperatures and pressures.

To put the technology in users’ hands, the researchers built an interactive tool, available online, in which anyone can select mineral attributes, have them converted automatically into a prompt, adjust parameters such as random seed, image size, guidance ratio and inference steps, and receive a watermarked generated image. The watermark is a deliberate safeguard against misuse and protects the integrity of the underlying dataset. The authors envision the tool serving mineralogists seeking a visual reference for attribute combinations, enthusiasts exploring the diversity of the mineral world, and science educators making abstract properties like cleavage and streak tangible for students. They are candid about limitations: synthetic images cannot be verified as depictions of real entities, the uniform description template may constrain image-text alignment, and only ten of Mindat’s more than 140 attributes were used, partly due to the text encoder’s input limits.

The road ahead is ambitious. The team proposes supplementing fine-tuning with supervised fine-tuning on higher-quality pairs and with preference-based optimization methods such as direct preference optimization to sharpen alignment between images and descriptions. They also sketch the inverse problem: fine-tuning a generative image-to-text model so that a photograph of an unknown specimen yields a description of its compositional and optical attributes, bypassing traditional classification pipelines in favor of reasoning from visual cues. And they suggest benchmarking against newer generators beyond Stable Diffusion. If mineral formation is a story of temperature, pressure, redox state and fluid chemistry unfolding over billions of years, a model that has learned the visual grammar of that story may one day help predict what minerals on other planets look like, or what fleeting intermediate phases in a crystal’s birth might have been, long before any camera could capture them.

Subject of Research: Fine-tuning the Stable Diffusion text-to-image model with Mindat mineral data to generate mineral images from textual descriptions of attribute combinations.

Article Title: Fine-tune the stable diffusion model using mindat data to generate mineral images from textual descriptions of attribute combinations

Article References: Fu, Q., Que, X., Ma, X., Lin, M., Sun, S., & Chen, M. (2026). Fine-tune the stable diffusion model using mindat data to generate mineral images from textual descriptions of attribute combinations. Earth Science Informatics, 19(11), Article 190. https://doi.org/10.1007/s12145-026-02235-2

Image Credits: AI Generated

DOI: 10.1007/s12145-026-02235-2

Keywords: Mindat, Stable Diffusion, text-to-image generation, mineralogy, mineral informatics, fine-tuning, LoRA, OpenMindat, generative AI, crystallography, image-text dataset, Earth Science Informatics