Traditional segmentation models operate under what the authors call the closed-world assumption: they can only label the fixed set of categories seen during training. According to the survey, this limitation makes them fragile in dynamic environments where conditions change and previously unknown objects emerge, which matters for critical applications such as autonomous driving and medical imaging.

To address these weaknesses, the field has shifted toward open-ended settings, and the survey divides this shift into three distinct research directions. Open-Set Segmentation focuses on the binary discrimination of unknown classes, deciding whether a pixel belongs to a known category or not. Open-Vocabulary Segmentation leverages large-scale Vision-Language Models to achieve zero-shot generalisation through natural language prompts. Open-World Segmentation goes further, emphasising autonomous discovery of new categories and incremental knowledge maintenance as systems keep learning.

A central contribution of the paper is a unified reference framework that spans these three research categories, alongside a summary of the common datasets and evaluation metrics used across the field. The authors also systematically classify key methodological routes, covering uncertainty estimation, vision-language alignment and knowledge distillation as the main technical pillars supporting open-ended segmentation.

Beyond taxonomy, the survey provides an in-depth analysis and quantitative performance comparisons of state-of-the-art techniques in these domains. This comparative element is designed to help readers see where current methods stand relative to one another, rather than simply listing approaches in isolation.

The paper also identifies the prevalent challenges holding the field back. Among them are computational efficiency, semantic ambiguity, and the lack of unified benchmarks, an issue that complicates fair comparison between methods developed under different assumptions and evaluation protocols.

Looking forward, the authors outline potential research trajectories intended to guide subsequent technological development and innovation. The collected methods and datasets referenced in the survey are being made publicly available through a dedicated GitHub repository, Awesome-Open-ended-Segmentation, so that other researchers can build on the compiled resource.

For the broader computer vision community, the survey arrives at a moment when segmentation is increasingly entangled with large vision-language models and open-world learning. By consolidating definitions, datasets, metrics and method families into a single reference, the authors position the work as a valuable resource for researchers entering the area and for those tracking how open-ended scene understanding continues to evolve.

Subject of Research: A systematic survey of open-ended visual segmentation, covering open-set, open-vocabulary and open-world segmentation methods, datasets and challenges.

Article Title: A systematic survey of open-ended visual segmentation: taxonomies, challenges and future directions

Article References: Wang, A., Ren, C., Du, X., Wu, J., Zhang, X., & Liu, Y. (2026). A systematic survey of open-ended visual segmentation: taxonomies, challenges and future directions. Artificial Intelligence Review. https://doi.org/10.1007/s10462-026-11720-5

Image Credits: AI Generated

DOI: 10.1007/s10462-026-11720-5

Keywords: computer vision, visual segmentation, open-set segmentation, open-vocabulary segmentation, open-world segmentation, vision-language models, zero-shot generalisation, uncertainty estimation, knowledge distillation, autonomous driving, medical imaging, survey