The research team, led by Jiayu Lu and colleagues at the University of New Hampshire, identified a fundamental bottleneck in AAC clinical practice. Speech-language pathologists need quantitative metrics such as selection rate and type-token ratio to assess intervention progress, but calculating these metrics requires knowing whether each audio output from the device was triggered by the AAC user or by the communication partner who is modeling language. A national survey of school-based speech-language pathologists cited in the study found that the time required for transcription and attribution was one of the major barriers to routine use of language sample analysis. Without large-scale, accurately attributed data, evidence-based AAC intervention remains more aspiration than reality.

Previous attempts at automation have stumbled on this attribution problem. Automated data logging protocols such as the Language Activity Monitor record every device activation with a timestamp, and automatic speech recognition systems have been explored for transcription, but neither can determine who triggered the activation. Computer-vision solutions, such as a smart-glasses system that tracks hand gestures with egocentric cameras, face their own obstacles: continuous video recording raises privacy and consent concerns in homes and classrooms, hand occlusion can hide the user’s gestures, changing lighting conditions degrade performance, and requiring the AAC user to wear additional hardware can be uncomfortable or distracting for people with sensory sensitivities.

The smart ring system sidesteps all of these issues with an elegant asymmetry. Only the communication partner wears hardware — a 3D-printed ring housing two circuit boards, a 6-axis inertial measurement unit sampling acceleration and gyroscope data at 120 Hz, an ESP32-H2 microcontroller transmitting via Bluetooth Low Energy, and a 40 mAh battery. The AAC user wears nothing at all, eliminating any physical or cognitive burden. Meanwhile, the smartphone’s microphone records the ambient audio at 44.1 kHz. The attribution logic exploits a simple temporal fact: every AAC audio output is triggered by a screen tap. If a recognized AAC audio event coincides with a detected tap from a ring-wearing partner, the audio belongs to that partner; if not, it belongs to the user.

The audio processing pipeline works in two stages. First, a voice activity detection module resamples incoming audio to 16 kHz, applies an 80 Hz high-pass filter to remove low-frequency rumble, and uses a real-time detection model to isolate speech segments, retaining only those longer than 150 milliseconds and padding each segment by 50 milliseconds to preserve initial and final phonemes. Second, each isolated segment is passed through a pre-trained ECAPA-TDNN speaker-recognition network that converts it into a 192-dimensional embedding vector capturing its acoustic characteristics. A support vector machine then performs the binary classification: AAC device audio versus everything else, including human speech. Because the system only needs to identify device-generated audio rather than transcribe individual voices, the task is far more tractable than full speech recognition.

Tap detection posed a subtler engineering challenge. The team discovered that the delay between a screen tap and the resulting audio is not fixed — it follows a bimodal distribution, with one cluster averaging 0.30 seconds and another 0.61 seconds, likely reflecting whether the device retrieves audio from memory or from disk. Rather than assuming a fixed offset, the algorithm searches a window spanning 0.1 to 1.0 seconds before each detected audio onset, looking for the peak in jerk — the rate of change of acceleration. Jerk is the key discriminator: smooth arm movements produce broad acceleration curves, while a physical tap against a screen creates an almost instantaneous collision signature. Once the jerk peak and an adjacent local minimum are located, a 20-sample window of IMU data is extracted and fed into a one-dimensional convolutional neural network that classifies it as tap or non-tap.

The results from a pilot study with 18 healthy adults simulating AAC interactions were striking. Voice activity detection caught 98.89 percent of speech events. The audio classifier achieved 99.83 percent accuracy on the training dataset and 99.64 percent on the test dataset under rigorous leave-one-subject-out and leave-one-group-out validation schemes designed to prevent data leakage. The full cascaded pipeline — voice detection, audio classification, tap detection, and attribution — maintained a cumulative end-to-end accuracy of 94.81 percent across 1,620 test events. Notably, a model trained only on normal tap patterns performed comparably to one trained on light, normal, and firm taps, and errors were distributed almost evenly between the two possible directions of misattribution, meaning the system neither systematically inflates nor deflates the user’s measured performance.

Beyond attribution, the system automatically computes three clinically meaningful metrics. Selection rate measures the user’s true information transfer speed in bits per second, isolated from the partner’s modeling inputs that artificially inflate conventional log-based counts. A newly proposed modeling balance index captures the percentage of total device activations attributable to the user, allowing clinicians to track how the balance of use shifts across sessions and adjust the amount of partner modeling accordingly. And a user-specific type-token ratio measures genuine lexical diversity by filtering out the advanced vocabulary that teachers model during sessions — a correction that prevents artificial inflation of the user’s apparent vocabulary breadth. Mean absolute percentage errors for these metrics ranged from 6.66 to 7.39 percent, though the authors caution that with only six test groups, these intervals represent uncertainty estimates from a pilot dataset rather than clinical-grade measurement accuracy.

The design also delivers a remarkable computational efficiency gain. Because tap detection is triggered only when an AAC audio event is detected, and each activation processes just a single 20-sample window, the system analyzed only a small fraction of the continuously recorded IMU stream — a 93.1 percent reduction in computational load compared with a traditional sliding-window approach. This efficiency matters for real-world deployment on consumer smartphones, where battery life and processing overhead constrain what wearable sensing systems can realistically sustain throughout a full therapy day.

Significant limitations remain before the system can leave the laboratory. The pilot involved healthy adults in a controlled environment with a vocabulary of only ten isolated words from a single device and synthesized voice; real interventions involve overlapping speech, variable tapping forces, ambient noise, phrases and sentences, and multiple devices. When caregivers speak over the device — as they often do, verbally confirming a word precisely as they activate it — the audio classifier may struggle, and the authors propose blind source separation algorithms and smartphone multi-microphone beamforming as future solutions. Only two communication partners were tested, and confidence-based conflict resolution was exercised in just four cases. Still, the core demonstration stands: a privacy-preserving, video-free, user-unburdened sensing platform that attributes every device activation with over 94 percent accuracy. If future work validates it in authentic clinical settings, the humble smart ring could transform AAC therapy from a data-starved discipline into a data-rich one, finally enabling the large-scale, evidence-based intervention research that millions of users with complex communication needs have been waiting for.

Subject of Research: A smart-ring-based multimodal sensing and machine learning system for automated data collection, attribution, and analysis in augmentative and alternative communication intervention.

Article Title: Machine learning-driven multimodal sensing for automated data collection, attribution, and analysis in augmentative and alternative communication

Article References: Lu, J., Ghoreishi, N., Chen, S.-H. K., & Chen, D. (2026). Machine learning-driven multimodal sensing for automated data collection, attribution, and analysis in augmentative and alternative communication. Machine Learning with Applications, 26, Article 101022. https://doi.org/10.1016/j.mlwa.2026.101022

Image Credits: AI Generated

DOI: 10.1016/j.mlwa.2026.101022

Keywords: augmentative and alternative communication, smart ring, wearable sensing, machine learning, inertial measurement unit, voice activity detection, convolutional neural network, speech-generating devices, attribution, speech-language pathology, ECAPA-TDNN, clinical metrics