Smart Glove and Tiny Transformer Read Multi-Digit Sign Language Numbers in Real Time

For the millions of deaf and hard-of-hearing people worldwide, everyday interactions that most people take for granted—giving a phone number, quoting a price, dictating an address—can become frustrating bottlenecks when no interpreter is available. Camera-based sign language recognition systems have made impressive strides in recent years, but they come with baggage: they demand significant computing power, they raise privacy concerns when cameras must watch hands and faces continuously, and they struggle in poor lighting or cluttered environments. A new study published in Multimedia Tools and Applications by Ayoub Parvizi, Kamal Jamshidi, and Hamed Shahbazi of the University of Isfahan in Iran offers a strikingly lean alternative. Their system recognizes continuous sequences of multi-digit numbers in Iranian Sign Language using nothing more than a wearable glove fitted with six inertial measurement units, paired with a compact transformer model light enough to run on edge devices.

The heart of the contribution is ISL-MDN, a publicly released dataset of inertial recordings capturing continuous multi-digit number signing. Forty-one participants wore a glove instrumented with six IMU sensors sampling at 100 hertz, producing streams of acceleration and angular velocity data as they signed sequences of two, three, and four digits. Crucially, the dataset includes sign boundary annotations, meaning researchers know exactly where one digit sign ends and the next begins. This annotation scheme allows the systematic evaluation of two fundamentally different recognition paradigms within a single benchmark: segment-based approaches that classify each pre-segmented sign individually, and sequence-based approaches that must infer the digit string from the unbroken sensor stream. The researchers also reserved unseen six-digit sequences specifically to test whether models trained on shorter sequences could generalize to longer, previously unencountered combinations—a rigorous test of true sequence understanding rather than memorization.

The choice of inertial sensing is deliberate and consequential. Unlike video, IMU data contains no images of the signer, so privacy is preserved by design; there is nothing to identify a face, a location, or a bystander. The data volume is also dramatically smaller. The authors calculate that, normalized by the number of represented classes, ISL-MDN requires more than fifty times less storage per class than comparable video-based datasets. That compactness matters enormously for edge AI, where memory, bandwidth, and battery budgets are tight. A glove with six tiny motion sensors is cheap, unobtrusive, and works identically in a dark room or bright sunlight—conditions that routinely defeat camera systems.

On the algorithmic side, the team benchmarked architectures spanning both recognition paradigms. For segment-based recognition, they built a ResNet convolutional baseline and a more sophisticated model they call the Transformer Core Method, or TCM. TCM exploits the transformer architecture’s celebrated attention mechanism in two directions at once: it models dependencies between the six sensor channels within each IMU frame, capturing how the motion of one part of the hand relates to another at a single instant, and it models temporal relationships across the segmented sign units, capturing how the hand’s configuration evolves from the start to the end of a digit. This dual attention lets the network weigh which sensor channels and which moments in time carry the most discriminative information for distinguishing, say, a signed four from a signed five.

Segment-based recognition, however, sidesteps the hardest part of the problem: in the real world, nobody tells the system where one sign ends and the next begins. For sequence-based recognition, the authors proposed their centerpiece, the Hierarchical Temporal Compression Transformer, or HTCT. Continuous sign language recognition from raw sensor streams is notoriously difficult because the model must implicitly align an input sequence of arbitrary length with an output label sequence of unknown length, without frame-level supervision. HTCT tackles this with a training objective based on Connectionist Temporal Classification, or CTC, a technique originally developed for speech recognition that allows the network to learn the alignment itself by summing over all possible alignments between inputs and labels.

The truly novel ingredient in HTCT is its hierarchical temporal compression. Raw IMU streams at 100 hertz are highly redundant: consecutive samples carry largely overlapping information, and the discriminative structure of a signed digit unfolds over hundreds of milliseconds, not milliseconds. HTCT combines a fixed Haar wavelet decomposition—a classical signal processing tool that decomposes a signal into coarse trends and fine details at multiple scales—with a learnable, causal temporal compression module. The wavelet stage provides a principled multi-scale representation, capturing both the rapid transients of a hand flick and the slower posture changes that define a digit, while the learnable compression stage is trained to discard redundant temporal information and retain only the features that help the CTC objective. The result is a much shorter, information-dense sequence that the transformer can attend over efficiently, improving both accuracy and speed.

The numbers tell a compelling story. The ResNet baseline achieved a Word Error Rate of 16.3 percent, where a word corresponds to a signed digit and the error rate measures how often the recognized digit sequence differs from the ground truth. TCM improved on that with 14.6 percent, confirming the value of modeling cross-channel dependencies with attention. But HTCT left both far behind, reaching a Word Error Rate of just 5.2 percent—roughly a threefold reduction in errors relative to the baseline. Even more remarkable is what that accuracy costs computationally: only 1.73 million floating-point operations per inference. For context, that is orders of magnitude below what typical video-based sign language transformers require, placing HTCT squarely in the territory of microcontrollers and low-power wearable processors. The authors note that the model achieves an effective accuracy–efficiency trade-off through low computational cost and inference latency, while candidly acknowledging that power consumption was not directly measured in the study.

The generalization test adds further weight. When confronted with six-digit sequences—longer than anything seen during training—the system’s ability to recognize previously unseen combinations demonstrates that HTCT is learning compositional structure rather than memorizing fixed-length patterns. This matters because real-world usage is inherently open-ended: a signer might need to communicate a bank account number, a serial code, or a date of arbitrary length. A recognizer that only works for the sequence lengths in its training set would be of limited practical value; one that composes digit signs flexibly can scale to the task at hand.

The release of ISL-MDN as an open resource, hosted on Zenodo with source files and documentation on GitHub, may prove as influential as the model itself. Inertial sensing remains relatively underexplored for continuous sign language recognition compared with vision, and one persistent obstacle has been the scarcity of annotated, signer-diverse datasets. By providing boundary annotations, multiple sequence lengths, held-out generalization data, and forty-one participants, ISL-MDN gives the community a unified benchmark on which segment-based and sequence-based methods can be compared fairly. It also extends coverage to Iranian Sign Language, a language underrepresented in the predominantly English- and Chinese-centric sign language recognition literature, and the data collection followed institutional ethical guidelines with informed consent and full anonymization of participant data.

The broader significance lies in what this work suggests about the future of assistive wearables. Sign language recognition has long been framed as a computer vision problem, but the wrist and fingers are where the signal actually lives, and miniature motion sensors can capture it directly, privately, and cheaply. Combined with aggressively compressed transformer architectures like HTCT, the vision of a glove that translates continuous signing into text entirely on-device—no cloud, no camera, no privacy trade-off—moves from aspiration toward engineering reality. Challenges remain, including scaling beyond digits to full vocabularies, testing across more sign languages, and validating battery life in the field. But with a 5.2 percent error rate at 1.73 million FLOPs, this study makes a persuasive case that the next breakthrough in accessible communication may come not from bigger models watching us, but from smaller ones riding along on our hands.

Subject of Research: Inertial-sensor-based continuous sign language number recognition using transformer models for edge AI

Article Title: Inertial-sensor–driven continuous multi-digit sign language number recognition using hierarchical temporal compression and transformer-based methods for edge AI

Article References: Parvizi, A., Jamshidi, K., & Shahbazi, H. (2026). Inertial-sensor–driven continuous multi-digit sign language number recognition using hierarchical temporal compression and transformer-based methods for edge AI. Multimedia Tools and Applications, 85(10), Article 792. https://doi.org/10.1007/s11042-026-21947-7

Image Credits: AI Generated

DOI: 10.1007/s11042-026-21947-7

Keywords: sign language recognition, inertial sensors, wearable technology, transformer, edge AI, IMU, CTC, wavelet decomposition, Iranian Sign Language, dataset, deep learning, accessibility