Compact Transformer Learns Kazakh Sign Language From Landmarks, Not Pixels

Researchers in Kazakhstan have built a compact artificial intelligence model that can recognize isolated gestures in Kazakh Sign Language (KzSL) from people it has never seen before—a notoriously difficult test for any gesture-recognition system. The work, published in Frontiers in Artificial Intelligence, tackles a problem that has long sidelined most of the world's sign languages: while American, British, and German Sign Language enjoy large annotated datasets and mature benchmarks, KzSL has had almost no computational resources at all. The team's answer is not a massive video model but a deliberately small Transformer that reads skeletal landmarks instead of raw pixels, and it outperformed three well-known baseline architectures on every test partition in a strict signer-independent evaluation.

The starting point was data—or rather, the lack of it. The researchers recorded a purpose-built corpus of KzSL gestures in a controlled studio with green screens, professional lighting, and GoPro HERO 12 cameras shooting at 4K resolution. Five signers performed each gesture roughly 20 to 25 times, producing 39,979 samples spread across 187 output classes: 186 gesture classes plus one "no_event" class representing stretches of video without a target gesture.

Rather than feeding raw video into a heavy neural network, the team used MediaPipe Holistic to extract landmarks from every frame. Each frame became a 1,662-dimensional vector: 33 pose landmarks described by four values each, 468 face landmarks, and 21 landmarks for each hand described by three values apiece. Every gesture clip was then resampled to a fixed length of 56 frames, z-score normalized within the clip itself, and augmented during training with Gaussian coordinate noise and random frame dropout.

An automated segmentation pipeline, based on a motion score computed from pose and hand landmarks, isolated the active phase of each gesture, with manual review layered on top to check the automated boundaries. The architecture itself, called SignTransformerModel, contains about 3.5 million trainable parameters—small by modern deep learning standards. Its defining feature is a multi-branch input stem.

Instead of projecting the entire 1,662-dimensional frame vector through a single linear layer, the model splits the input into four anatomical streams—pose, face, left hand, and right hand—and projects each through its own branch before fusing them into a unified 256-dimensional representation. The design is motivated by the linguistics of signing: hands carry the primary manual articulation, posture provides spatial context, and the face contributes non-manual markers such as expression and mouth movement. The split also prevents a practical pitfall, since face landmarks account for 1,404 of the 1,662 input dimensions and could otherwise drown out the low-dimensional but highly discriminative hand features.

After the stem, sinusoidal positional encodings restore the temporal order that self-attention alone cannot perceive, and a four-layer Transformer encoder with eight attention heads models dependencies across the 56-frame sequence—from the preparation of a sign through its stroke to its retraction. A TemporalAttentionPool module then compresses the sequence into a single vector by combining learned attention pooling with max pooling, preserving the most salient activations rather than averaging them away. A lightweight classification head with layer normalization and dropout maps the result to 187 logits.

Evaluation was deliberately unforgiving. The team used a nested, signer-independent Leave-One-Signer-Out cross-validation: for each of the five partitions, one signer's data was withheld entirely from both training and model selection, the number of training epochs was chosen by inner cross-validation on the remaining four signers, and the refitted model was evaluated exactly once on the held-out signer. This corrected a leakage problem in the simpler single-level protocol, where the held-out signer's data had influenced epoch selection.

The stakes are real: under a conventional random split, where samples from the same signer can appear on both sides of the divide, preliminary experiments reached around 87 percent top-1 accuracy—an optimistic figure that mostly reflects the model memorizing individual signing styles rather than learning signer-invariant gesture representations. Under the strict protocol, the multi-branch Transformer achieved a mean accuracy of 0.6698, a mean macro F1 of 0.4207, and a mean weighted F1 of 0.6546 across the five outer test partitions. It beat all three baselines—a standard single-stem Transformer, a Temporal Convolutional Network, and a Bidirectional GRU—in macro F1 on every partition.

The comparison is informative because the standard Transformer shares the proposed model's temporal core and pooling but lacks the anatomical branch separation, so the performance gap points to the value of processing body parts separately. The TCN came closest, while the BiGRU trailed the field. Still, the spread across partitions was wide: macro F1 ranged from 0.3719 to 0.4897, a reminder that with only five signers, inter-signer variability in amplitude, tempo, and hand dominance remains a major source of error.

The headline numbers also deserve a careful reading, and the authors are candid about this. The no_event class makes up 42 percent of the corpus—16,794 samples—and every architecture recognizes it with near-ceiling F1, around 0.95 or higher. That inflates aggregate accuracy and weighted F1.

When accuracy is restricted to true gesture samples only, the proposed model's score drops from 0.6698 to 0.4490, a fall of more than 22 percentage points. Macro F1, which weights all 187 classes equally, tells the harsher story: many rare and difficult gestures drag it down even as the dominant non-gestural class props up the support-weighted metrics. Class-level analysis revealed two distinct failure mechanisms.

The first is structural under-representation. The gesture Mumkin was recorded by only three of the five signers and is entirely absent from the other two, so in each test partition the model learns it from just two signing styles. Its F1 was 0.000—consistently, across all four architectures, which strongly suggests a data problem rather than a modeling flaw.

Twenty-nine of the 186 gesture classes have incomplete signer coverage, and the gaps cluster around specific signers' availability rather than being randomly distributed. The second mechanism is systematic confusion between visually or articulatorily similar gestures: the most frequent error pairs included Memleket/Ul, Qorqu/Turu, Tabigat/Aua raiy, Dayindau/Jol, and Ote/Sondyqtan. Some of these persisted across both evaluation protocols, and the authors note that resolving them will likely require expert review and targeted re-recording rather than more training epochs.

The study has clear limitations that the authors spell out: five signers cannot capture the diversity of a signing population; the pipeline depends on the accuracy of the upstream landmark extractor and loses appearance-based cues that raw video would provide; the evaluation is offline, so real-time latency has not yet been benchmarked on identical hardware; and the baseline comparison rests on a single random seed, with the TCN trained under non-deterministic settings. Yet the core finding stands: a compact, landmark-based Transformer is a viable recognition component for a low-resource sign language, and the path forward is as much about data—more signers, full class coverage, expert-guided review of confusable gestures—as about architecture. If the team's roadmap holds, the model could eventually anchor KzSL dictionaries, educational tools, and assistive communication systems, developed, as the researchers emphasize, with the direct involvement of the deaf community itself.

Subject of Research: A lightweight multi-branch Transformer for signer-independent recognition of Kazakh Sign Language from MediaPipe Holistic landmarks

Article Title: A lightweight multi-branch transformer architecture for Kazakh Sign Language recognition under signer-independent conditions

Article References: Amangeldy, N., Yerimbetova, A., Zhamangarin, D., Berzhanova, U., Kurmetbek, B., & Gazizova, N. (2026). A lightweight multi-branch transformer architecture for Kazakh Sign Language recognition under signer-independent conditions. Frontiers in Artificial Intelligence, 9, 1947364. https://doi.org/10.3389/frai.2026.1947364

Image Credits: AI Generated

DOI: 10.3389/frai.2026.1947364

Keywords: Kazakh Sign Language, sign language recognition, Transformer, MediaPipe Holistic, landmark-based recognition, signer-independent evaluation, low-resource languages, deep learning, cross-validation, temporal attention pooling, class imbalance, assistive technology