Simple Machine Learning Outperforms Deep Models in Somali Dialect Test

Researchers have built one of the first text datasets for identifying Somali’s two primary dialects, MAXAA TIRI and MAAY, and tested how well a range of machine learning and deep learning models can tell them apart. The study, published in Discover Data, reports that a simple Naive Bayes classifier reached 99.05 percent accuracy under controlled test conditions, narrowly beating more complex architectures including bidirectional long short-term memory networks, which peaked at 98 percent.

Somali is spoken by more than 22 million people, primarily in Somalia and Djibouti and among diaspora communities, and it shows substantial dialectal variation. Most existing Somali natural language processing treats the language as a single entity, and even Somali-specific transformer models such as SomBERTa have been trained exclusively on written MAXAA TIRI text, leaving MAAY, an oral-first dialect with inconsistent orthography, largely outside the reach of modern language technology.

To address that gap, the team assembled 8,947 manually annotated Somali text samples drawn from heterogeneous sources: social media posts and comments on Facebook, news material from outlets including BBC and VOA, blogs, books, and formal documents such as the Somali-language version of the Southwest State Constitution. The split was uneven, with 5,505 samples, or 61.5 percent, in MAXAA TIRI and 3,442 samples, or 38.5 percent, in MAAY.

Annotation was carried out by three native Somali linguists, two specializing in MAXAA TIRI and one in MAAY, who labeled each sample independently following a guideline based on dialect-specific lexical, morphological, and orthographic indicators. Ambiguous cases, which frequently involved code-switching, shared vocabulary, borrowed Arabic terms, and informal social media expressions, were resolved through consensus discussion. Inter-annotator agreement measured with Cohen’s Kappa reached 0.92, indicating high reliability.

A crucial caveat runs through the entire study: because naturally occurring written MAAY is scarce, 562 MAAY samples were produced by expert-guided translation from MAXAA TIRI texts by a dialectologist and native MAAY speaker. The authors state plainly that this may introduce partial structural parallelism between the two classes, meaning classification performance may be optimistic and all reported results should be interpreted as upper-bound estimates rather than definitive real-world performance.

The team applied several safeguards against overfitting and source bias. Duplicate and near-duplicate samples were removed before splitting, and the data were partitioned in a source-aware manner so that text from the same platform or document type never appeared in both training and test sets. The dataset was divided 80 percent for training and 20 percent for testing with stratified sampling, and ablation experiments showed that removing preprocessing steps such as stop-word removal and normalization degraded performance, suggesting the models relied on linguistically meaningful patterns rather than memorization.

On the modeling side, four traditional classifiers were trained with TF-IDF features and tuned via grid search: Naive Bayes, support vector machines, random forests, and logistic regression. Naive Bayes led with 99.05 percent accuracy, followed by SVM at 98.55 percent, logistic regression at 97.49 percent, and random forest at 94.75 percent. Six deep learning architectures were then trained on tokenized, padded sequences using TensorFlow and Keras with the Adam optimizer: BiLSTM and CNN tied for the best deep results at 98 percent accuracy, LSTM and MLP reached 97 percent, CNN-LSTM achieved 96.93 percent, and a plain RNN trailed at 90.84 percent.

Sevenfold cross-validation on the training data reinforced the picture of internal stability, with Naive Bayes achieving a mean accuracy of 99.65 percent and a standard deviation of just 0.13 percent across folds. The authors are careful, however, to note that cross-validation was performed on the same dataset distribution, including the partially translated MAAY samples, so it does not reflect performance variability on naturally occurring, real-world dialect data. A deployed web-based demonstration of the trained Somali Dialect Predictor, built with Flask, is likewise described only as a feasibility showcase, not an independent validation on externally sourced data.

A qualitative error analysis identified four main sources of misclassification: short sentences with too little lexical information, vocabulary shared between the two dialects, code-switching and mixed dialect usage in informal text, and inconsistent spelling on social media. Some errors may also stem from translated MAAY samples retaining structural patterns of MAXAA TIRI. The authors caution that reported accuracy, particularly for Naive Bayes, may partially reflect dataset-specific artifacts rather than purely dialectal distinctions.

The broader takeaway is a trade-off between model complexity and data availability in low-resource NLP. The strong Naive Bayes result suggests Somali dialect identification is largely driven by discriminative lexical cues captured well by probabilistic models over TF-IDF representations, while the marginal gains of deep models indicate diminishing returns at this dataset size. Transformer-based models such as mBERT, XLM-R, and SomBERTa were excluded from evaluation because existing Somali-specific pretraining covers almost no MAAY, which would bias any comparison. The authors frame their contribution as a linguistically validated dataset and empirical insight into model behavior, not production readiness, and call for MAAY-inclusive pretraining resources, speech-based data, more organically produced MAAY text, and transfer learning from related Cushitic languages as future directions.

Subject of Research: Machine learning and deep learning approaches to identifying the Somali MAXAA TIRI and MAAY dialects in text

Article Title: Somali dialect identification in low-resource settings using machine learning and deep learning

Article References: Gedi, A. A., Ahmed, Y. M., Mohamed, S. A., Yusuf, Y. A., & Ebdiyow, A. U. (2026). Somali dialect identification in low-resource settings using machine learning and deep learning. Discover Data, 4(1), Article 10. https://doi.org/10.1007/s44248-026-00109-y

Image Credits: AI Generated

DOI: 10.1007/s44248-026-00109-y

Keywords: Somali language, dialect identification, natural language processing, low-resource languages, Naive Bayes, BiLSTM, machine learning, deep learning, text classification, MAAY dialect, MAXAA TIRI, dataset annotation