The clinical rationale is straightforward. Natural communication is rarely a single-stream activity. When people speak, they gesture constantly, and these cospeech gestures carry meaning that listeners integrate with the spoken words. Stroke and neurodegenerative diseases such as amyotrophic lateral sclerosis can strip away both speech and body movement at once, leaving patients dependent on slow, effortful typing systems. Prior BCI research had decoded speech, facial expressions, cursor control and manual gestures individually, but it remained unclear whether one implant could reliably decode speech and gestures when attempted together, as they so often are in everyday life.

The researchers worked with three participants enrolled in the BRAVO clinical trial, each implanted with a high-density electrocorticography, or ECoG, array containing 253 disk-shaped electrodes spaced three millimeters apart, placed subdurally over the left sensorimotor cortex. Bravo-1r and Bravo-3 had severe vocal-tract and limb paralysis from brainstem strokes, while Bravo-6 had severe vocal-tract paralysis and moderate limb impairment from ALS. Because the array spanned both precentral and postcentral gyri, it sampled motor and sensory representations for the face, hands, arms and head simultaneously, along with listening responses in the superior temporal gyrus.

Mapping the neural activity revealed a familiar but imperfect organization. Electrodes preferring hand and arm movements clustered on the dorsal-posterior aspect of the array, head and eye responses sat dorsally and anteriorly near putative frontal eye fields, and orofacial speech responses localized ventrally toward the Sylvian fissure. Yet the segregation was only partial. A subset of electrodes, particularly in the precentral gyrus, were strongly modulated during both speech and gestures. The team quantified this with an overlap score, the geometric mean of each electrode’s speech-related and gesture-related high-gamma activity, and found that high-overlap electrodes concentrated in the precentral gyrus in all participants tested.

That partial overlap mattered enormously for decoding. The researchers collected data in three behavioral contexts: speech-only trials, gesture-only trials, and simultaneous speech-plus-gesture trials in which participants attempted a target phrase and gesture concurrently. When neural network models were trained exclusively on isolated behaviors and then tested on simultaneous trials, performance degraded substantially. False negative rates, meaning genuine attempts misclassified as rest, climbed from near zero on isolated trials to as high as 34.9 percent for one participant when tested on simultaneous data. Conversely, models trained only on simultaneous data faltered on isolated trials. Decoder performance, in short, was context dependent.

The solution was context-inclusive training. Hybrid models trained on an equal split of isolated and simultaneous trials maintained high accuracy across both conditions, outperforming models trained on either data type alone. Gradient-based saliency analyses showed why: training on simultaneous behavior shifted the models’ reliance away from electrodes with high speech–gesture overlap and toward more separable, modality-specific signals. The overlapping electrodes were not simply discarded, however; many remained informative, just differently weighted. Analyses of the models’ internal embeddings showed that hybrid training brought the representations of the same phrase or gesture closer together across contexts, even though the geometry of isolated and simultaneous distributions never fully converged.

Crucially, the models did not need to memorize every possible phrase–gesture pairing. In a leave-one-out analysis with Bravo-6, who performed all 100 combinations of ten phrases and ten gestures, the researchers withheld all simultaneous trials containing one gesture from the gesture decoder’s training, or one phrase from the speech decoder’s, and found no significant accuracy difference between seen and unseen pairings. The gesture decoder achieved roughly 66 percent accuracy and the speech decoder roughly 80 percent on both conditions. This generalization to novel combinations is encouraging for scaling beyond proof-of-concept vocabularies, where exhaustively sampling every possible pairing would be impractical.

Running two decoders in parallel introduced another challenge: each model had to stay silent when the participant attempted the other modality. Models trained with only true-rest examples in their rest class produced false activations on 30.6 to 76.0 percent of opposite-modality trials. The fix was cross-modality training, in which each decoder’s rest class was built from a balanced mix of genuine rest and opposite-modality data labeled as rest. This drove opposite-modality false positive rates down to essentially zero while preserving low error rates on true rest. With both strategies combined, offline simultaneous decoding accuracy reached 68.8 percent for gestures and 77.5 percent for speech in Bravo-6, and 88.3 percent and 84.0 percent respectively in Bravo-1r, far above chance.

The payoff came in real time. Decoded speech appeared as on-screen text while decoded gestures animated a personalized full-body avatar, built in Unreal Engine’s MetaHuman Creator and refined through interviews with the participants about their preferred appearance and virtual environment. In a gesture-only copy task, Bravo-6 achieved 81.8 percent real-time accuracy; in the simultaneous task, 66.0 percent for gestures and 70.0 percent for speech. During cued conversation blocks, accuracy held at 85.0 percent for gestures and 75.0 percent for speech, and in a qualitative demonstration the participant’s avatar conversed with a second experimenter-controlled avatar, exchanging speech and gestures in a shared virtual space.

The authors are careful about limitations. Vocabularies were small, ipsilateral movements decoded less reliably than contralateral ones, and generalization to concurrent attempts remained imperfect, possibly reflecting the cognitive demands of simultaneous behavior and limited training data. With only three participants, differences in paralysis etiology and residual capability could not be fully disentangled. Still, the study establishes a practical recipe for multi-effector BCIs: sample broad sensorimotor territory with a single implant, train on both isolated and simultaneous behaviors, and use cross-modality rest training to keep parallel decoders honest. As the researchers conclude, these findings mark a concrete step toward unified neuroprosthetic systems that restore not just words, but the full, gestural richness of human communication.

Subject of Research: Simultaneous decoding of speech and gestures from a single electrocorticography brain–computer interface in people with paralysis

Article Title: Simultaneous speech and gesture decoding for multimodal communication in paralysis

Article References: Brosler, S. C., Liu, J. R., Silva, A. B., Hallinan, I. P., Kurtz-Miott, C. M., Dunkel Wilker, J. F. B., Tu-Chan, A., Ganguly, K., & Chang, E. F. (2026). Simultaneous speech and gesture decoding for multimodal communication in paralysis. Nature Neuroscience, 29(10), 2445-2455. https://doi.org/10.1038/s41593-026-02446-2

Image Credits: AI Generated

DOI: 10.1038/s41593-026-02446-2

Keywords: brain-computer interface, electrocorticography, speech decoding, gesture decoding, paralysis, sensorimotor cortex, neuroprosthesis, ALS, stroke, virtual avatar, neural decoding, Nature Neuroscience