Prompt Robustness and Fine-Tuning Tested in Open-Vocabulary Object Detection Showdown

Open-vocabulary object detection has quietly become one of the most consequential ideas in modern computer vision. Instead of being locked to a fixed list of categories learned during training, these models can recognize objects described in plain language—a text prompt such as “traffic sign” or “kitchen appliance” is enough to make them find and localize instances they were never explicitly trained on. The promise is enormous: robots that understand novel instructions, surveillance systems that adapt to new threats, and annotation pipelines that label images without human effort. Yet a new systematic study from Selcuk University in Konya, Türkiye, suggests that the field’s enthusiasm for headline accuracy numbers has obscured a more complicated reality, one in which the choice of words, the cost of inference, and the fate of unseen classes after fine-tuning can matter as much as raw performance.

The research, published in Multimedia Tools and Applications by Melisa Alara Ozuberk and Ilkay Cinar, delivers one of the first head-to-head evaluations of three leading real-time open-vocabulary detectors: YOLO-World, its successor YOLO-World v2, and YOLOE. Rather than benchmarking accuracy alone, the authors designed their experiments around three scenarios that mirror how these systems are actually deployed: zero-shot inference on entirely new datasets, sensitivity to variations in the textual prompts that steer detection, and fine-tuning on domain-specific data followed by tests of whether open-vocabulary generalization survives. The evaluation spans three datasets with deliberately different characteristics—the classic VOC2012 segmentation subset, the HomeObjects-3K indoor detection dataset, and the demanding KITTI autonomous driving benchmark.

The zero-shot results reveal a striking dependence on domain. The highest performance was achieved on HomeObjects-3K, where YOLO-World v2 reached 0.443 mAP@0.5:0.95, a metric that rewards both accurate localization and correct classification across a range of overlap thresholds. KITTI, by contrast, produced the weakest results across all three models, a consequence of domain shift: the driving imagery, with its unusual viewpoints, small distant objects, and harsh lighting conditions, differs substantially from the data distributions these models encountered during pre-training. On the VOC dataset, YOLOE claimed the highest zero-shot accuracy at 0.310 mAP@0.5:0.95, outperforming both YOLO-World variants on that benchmark, although this advantage came with a caveat that emerged clearly in the timing analysis.

That caveat is speed. YOLO-World v2 proved to offer the best overall balance between accuracy and throughput, sustaining between 20 and 30 frames per second—comfortably real-time for many applications. YOLOE, despite its stronger zero-shot accuracy on VOC, exhibited lower inference speed in some configurations, a trade-off that could prove decisive in latency-sensitive settings such as autonomous navigation or live video analytics. The YOLO-World family also benefited from an embedding cache mechanism, which pre-computes text embeddings for the prompt vocabulary and reuses them across frames. Latency analysis with increasing prompt counts showed that this design keeps inference efficient even as the number of textual categories grows, an architectural advantage that becomes more valuable the richer the vocabulary deployed in production.

Perhaps the most practically important finding concerns prompt robustness—the question of how much detection quality degrades when the words fed to the model change. The authors constructed four categories of prompts for each dataset: base prompts, attribute prompts that add descriptive modifiers, longer descriptive prompts, and noisy prompts containing degraded or perturbed language. The results showed measurable performance drops under noisy prompt conditions for all models, confirming that open-vocabulary detectors are not immune to the fragility of language interfaces that has been documented across the broader vision-language literature. However, YOLO-World v2 maintained better stability across these variations than its competitors, suggesting that its training recipe or text-encoding pathway confers a degree of resilience that practitioners should weigh when deploying systems in the hands of non-expert users who cannot be relied upon to craft optimal prompts.

Fine-tuning delivered the expected gains but also exposed an uncomfortable truth about what adaptation costs. After fine-tuning on each dataset, mAP scores improved for all three models, demonstrating that standard transfer learning techniques remain effective when open-vocabulary detectors are specialized to a target domain. But the fine-tuned models showed zero performance on some unseen categories—classes that were never part of the fine-tuning data. This is precisely the failure mode that open-vocabulary detection is supposed to prevent, and its appearance after adaptation indicates that the boundary between open and closed vocabulary is thinner than the field often assumes.

The divergence between the model families was especially pronounced here. The YOLO-World family retained some of its open-vocabulary generalization ability after fine-tuning, continuing to respond to textual prompts for categories outside the training set. YOLOE, under the fine-tuning protocol adopted in the study, exhibited closed-set-like behavior: its predictions remained insensitive to the evaluated prompt variations, effectively behaving as if the text interface had been switched off and the model had reverted to a conventional fixed-category detector. For teams choosing between these architectures, the implication is significant—fine-tuning YOLOE may buy accuracy on known classes at the price of the very flexibility that motivated choosing an open-vocabulary model in the first place.

The study’s methodology reflects a growing recognition that evaluation practices in this field have been too narrow. The authors note that existing research has focused mainly on accuracy metrics while prompt robustness, unseen class generalization, and computational costs are rarely assessed together. By combining confusion-matrix-based analysis, standard detection metrics such as mAP at multiple intersection-over-union thresholds, and latency profiling under varying prompt counts, the work offers a template for more honest benchmarking. The datasets themselves are all publicly available—VOC2012 from the PASCAL repository, KITTI from the KITTI Vision Benchmark Suite, and HomeObjects-3K through its original repository—making the evaluation pipeline reproducible by other groups.

The broader context makes these findings timely. Open-vocabulary detection builds on a lineage that runs from the original YOLO real-time detector through open-set recognition and open-world detection to caption-supervised methods and the CLIP-style vision-language models that supply the text-image alignment these detectors depend on. YOLO-World, introduced in 2024, brought this capability to real-time speeds, and YOLOE pushed the concept further with its “see anything” design. Applications documented in the literature now span automatic image annotation, number plate recognition, wildlife monitoring, medical imaging, underwater fish counting, robotic navigation, and anomaly detection in surveillance—domains where the ability to name new categories without retraining is transformative.

For practitioners, the study’s bottom line is that model selection should be a multi-dimensional decision. Accuracy, inference speed, prompt robustness, and unseen class generalization form a set of trade-offs that no single model dominates. YOLO-World v2 emerges as the most balanced option, combining competitive accuracy, real-time throughput, an efficient embedding cache, and the strongest prompt stability. YOLOE offers the best zero-shot accuracy in some settings but pays in speed and, critically, appears to surrender its open-vocabulary character when fine-tuned. As these systems move from research demos into safety-relevant deployments—self-driving perception, medical triage, industrial inspection—the lesson of this comparative study is that the questions worth asking about a detector extend well beyond its leaderboard score, reaching into how it behaves when the words change, the domain shifts, and the training data runs out.

Subject of Research: Comparative evaluation of prompt robustness, fine-tuning, and generalization in open-vocabulary object detection models

Article Title: Prompt robustness, fine-tuning, and Generalization in open-vocabulary object detection: a comparative study of YOLO-World, YOLO-World v2 and YOLOE

Article References: Ozuberk, M. A., & Cinar, I. (2026). Prompt robustness, fine-tuning, and Generalization in open-vocabulary object detection: a comparative study of YOLO-World, YOLO-World v2 and YOLOE. Multimedia Tools and Applications, 85(10), Article 767. https://doi.org/10.1007/s11042-026-21928-w

Image Credits: AI Generated

DOI: 10.1007/s11042-026-21928-w

Keywords: open-vocabulary detection, YOLO-World, YOLOE, zero-shot object detection, prompt sensitivity, fine-tuning, computer vision, mAP, inference latency, domain shift, vision-language models, object recognition