The challenge is linguistic as much as computational. Skill expressions are short, leaving little context for disambiguation; they are noisy, mixing canonical terms with abbreviations, product names, misspellings, and boilerplate; and in Kazakhstan’s IT ecosystem they are structurally trilingual. Russian remains the dominant working language of the labor market, English is the lingua franca of technology terminology—so product names, frameworks, and hard-skill terms appear in English regardless of the surrounding language—and Kazakh is the state language in which a growing share of curricula and official competency descriptions are written. A single competency routinely appears as an English technical term, a Russian descriptive phrase, and a Kazakh official formulation. A matcher that handles only one or two of these silently drops a large part of the signal.

General-purpose multilingual encoders such as multilingual Sentence-BERT, LaBSE, and distilled paraphrase models map text from many languages into a shared vector space where cosine similarity approximates semantic relatedness. But these models are trained predominantly on general-domain parallel and paraphrase data. When confronted with terse, specialized skill phrases—and especially with Kazakh, which is under-represented in their pre-training—their similarity estimates degrade, conflating unrelated skills and separating equivalent ones. Kazakh poses additional difficulties: it is an agglutinative Turkic language with rich morphology, so a single skill stem appears in many inflected surface forms that scatter across the embedding space; technical Kazakh borrows heavily from Russian and English, so a Kazakh skill phrase is frequently a Kazakh grammatical frame wrapped around a Russian or English technical core; and labeled semantic-similarity data for Kazakh are essentially absent.

The researchers first benchmarked three representation families on a purpose-built labeled set of 3,616 English skill pairs, each annotated as equivalent or non-equivalent. The multilingual Sentence Transformer paraphrase-multilingual-mpnet-base-v2 proved the strongest skill matcher by a wide margin, reaching an F1 score of 0.8917—about twelve points above averaged static Word2Vec vectors and nearly twenty points above a mean-pooled monolingual BERT encoder. Notably, contextualization alone did not help: vanilla BERT embeddings, not optimized for cosine geometry, actually underperformed static vectors on the in-domain skill set even though they were competitive on general semantic-textual-similarity benchmarks. The lesson is that the choice of base encoder must be validated on the actual target distribution—terse skill phrases—rather than on a general benchmark, and that the training objective matters more than contextualization per se.

With the strongest base encoder selected, the team assembled a trilingual paraphrase fine-tuning corpus of 9,336 anchor-positive skill pairs, in which each positive is a meaning-preserving variant of its anchor: a translation across the three languages, an abbreviation or expansion, a near-synonym, or an alternative surface form of the same competency. Because Russian is the dominant source language and the bridge for much Kazakh code-mixing, the corpus was augmented with 3,095 additional Russian-specific pairs, derived by paraphrase mining with back-translation between Russian and English and filtered to retain only pairs with cosine similarity above 0.85 under the base encoder. Cross-lingual pairs, in which the anchor and the positive are in different languages, account for 2,815 pairs—roughly thirty percent of the total—and are the specific mechanism by which Kazakh is drawn toward its Russian and English equivalents in the shared space.

Fine-tuning used the Multiple Negatives Ranking objective, a contrastive loss that requires only positive pairs and treats the other in-batch examples as negatives, eliminating the need for hand-curated hard negatives. Training proceeded as a three-stage curriculum over four epochs: a cross-lingual alignment stage on pairs with at least one non-English language, a Russian specialization stage on the augmentation pairs, and a joint consolidation stage on all trilingual pairs. Staging matters because a single shuffled pass lets the abundant high-resource pairs dominate the gradient and starve the low-resource language; interleaving the stages keeps the Russian and, by transfer, the Kazakh signal present throughout training. After each stage, the model was evaluated on all three similarity benchmarks under both cosine and dot-product functions, and the checkpoint retained was the one achieving the highest mean of Russian cosine Pearson and Kazakh dot-product Pearson correlation.

The results show a fine-tuning regime that redistributes capacity toward where the base model was weakest. English performance was preserved, with correlations essentially unchanged, confirming that the staged schedule avoids catastrophic forgetting. Russian cosine Pearson correlation increased from 0.8125 to 0.8221, and Russian dot-product Spearman rose from 0.696 to 0.736. Kazakh improved from a much lower base: cosine Pearson rose from 0.5989 to 0.6050 and, most notably, dot-product Pearson rose from 0.4487 to 0.4912—a relative gain of roughly 9.5 percent, the largest relative improvement in the study. The Kazakh dot-product gain was statistically significant across five independent fine-tuning runs, with a bootstrap 95 percent confidence interval of [0.0381, 0.0469]. Because Kazakh received no dedicated monolingual similarity supervision, its improvement is evidence consistent with cross-lingual transfer rather than in-language fitting—the central low-resource finding of the paper.

An ablation study confirmed the mechanism. Removing cross-lingual pairs produced the largest single-component degradation in Kazakh dot-product Pearson, dropping from 0.4912 to 0.4601; removing the staged curriculum cost 0.0178; and removing Russian augmentation cost 0.0100. The additive decomposition suggests the components are largely complementary, with cross-lingual pairs providing the dominant signal and staged training providing an independent regularization benefit by preventing the high-resource language from dominating the gradient. The study also revealed an operationally meaningful asymmetry between similarity functions: fine-tuning re-scales embedding magnitude, so dot-product correlations move more than cosine ones, and the compressed Kazakh similarity distribution means a threshold tuned on English will systematically under-match Kazakh pairs—motivating per-language threshold calibration as the correct deployment choice.

The intrinsic gains carried over into a concrete downstream artifact. The team built a data-driven skill taxonomy by clustering 3,047 unique skill expressions extracted from 5,248 IT vacancies spanning 25 professions and 177 localities, collected from a national recruitment platform over February and March 2024. Agglomerative hierarchical clustering with average linkage outperformed K-Means, DBSCAN, and HDBSCAN, because skill embeddings form many small, tight communities rather than a few large blobs. After fine-tuning, the optimal distance threshold rose from 0.29 to 0.37, the number of skill communities grew from 1,121 to 1,407 finer-grained clusters, the mean silhouette improved from 0.2713 to 0.2851, and—most importantly—the share of erroneous clusters merging semantically distinct skills fell sharply from 19.74 percent to 13.97 percent. The singleton rate rose from 19.17 to 24 percent, an expected precision-coverage trade-off: as the space becomes more discriminative, more idiosyncratic skill strings stand apart rather than being absorbed into a loosely related cluster, and a clean singleton is preferable to a wrong merge in a taxonomy.

The error analysis identified four recurring failure types—abbreviation and product naming, code-mixing, over-generic boilerplate, and fine-grained sense distinctions—that cluster disproportionately in the low-resource language, where an abbreviation embedded in a Kazakh grammatical frame combines several problems at once. This concentration is encouraging for data collection, because a relatively small, well-targeted set of Kazakh paraphrase pairs covering common borrowed cores, their frequent abbreviations, and recurring boilerplate frames would address several error types simultaneously. The researchers released the trilingual paraphrase corpus, the labeled English evaluation set, and the fine-tuned encoder under permissive licenses, together with pinned dependency versions and fixed random seeds enabling exact reproduction. The team conjectures that the same recipe—choose a similarity-shaped multilingual base encoder validated on the target distribution, fine-tune contrastively with cross-lingual positives staged so the low-resource language is not drowned out, and calibrate the decision threshold per language—may generalize to any pipeline that must match short, specialized, multilingual expressions, from product catalogs to medical and legal terminology, though empirical validation in other domains and language portfolios is required before such claims can be made with confidence.

Subject of Research: A study reports fine-tuning a multilingual sentence transformer on a trilingual paraphrase corpus to improve low-resource skill-concept matching across Kazakh, Russian, and English.

Article Title: Fine-tuning multilingual sentence transformers for low-resource skill-concept matching across Kazakh, Russian, and English

Article References: Serikbayeva, S., Sambetbayeva, M., Ramazanova, V., Yerimbetova, A., Lamasheva, Z., Sadirmekova, Z., Batyrkhanov, A., & Mailybayev, Y. (2026). Fine-tuning multilingual sentence transformers for low-resource skill-concept matching across Kazakh, Russian, and English. Frontiers in Big Data, 9, Article 1900836. https://doi.org/10.3389/fdata.2026.1900836

Image Credits: AI Generated

DOI: 10.3389/fdata.2026.1900836

Keywords: sentence transformers, multilingual embeddings, Kazakh NLP, low-resource languages, contrastive fine-tuning, skill matching, semantic textual similarity, cross-lingual transfer, labor-market text mining, skill taxonomy, multiple negatives ranking loss, Frontiers in Big Data