Diffusion AI generates synthetic dysarthric speech to sharpen intelligibility classifiers

Researchers have built a generative artificial intelligence pipeline that synthesizes artificial dysarthric speech to boost the performance of deep learning systems that grade how intelligible a person’s impaired speech is. The study, published in Medical & Biological Engineering & Computing, trains a diffusion-based text-to-speech model called Grad-TTS on recordings from eight dysarthric speakers in the TORGO dataset, then uses that model to generate the entire 765-word vocabulary of a second corpus, UA-Speech, producing 6,120 synthetic utterances that carry the pathological vocal traits of the TORGO speakers while covering the lexical content of UA-Speech.

Dysarthria is a group of neurological speech disorders marked by abnormalities in the strength, speed, range, stability, tone or precision of the movements needed for respiration, phonation, resonance, articulation and prosody. It arises in conditions including amyotrophic lateral sclerosis, Parkinson’s disease and cerebral palsy, and it can leave speech slow, monotone, poorly articulated or broken by irregular pauses and breathy voice quality. Clinical assessment still relies on auditory-perceptual analysis by speech-language pathologists using standardized tools such as the Robertson Dysarthria Profile and the Frenchay Dysarthria Assessment, but that gold standard introduces inter-rater variability and limits reproducibility, which has motivated deep learning tools that classify intelligibility directly from acoustic features.

The central obstacle is data. The two benchmark datasets in this field, TORGO and UA-Speech, share well-known limitations: few speakers, limited phonetic diversity and under-representation of clinical variability. TORGO contains roughly 15 hours of speech from eight dysarthric individuals with cerebral palsy or ALS, age- and gender-matched to control participants, recorded simultaneously by an array microphone and a head-mounted microphone. UA-Speech holds around 102.7 hours of annotated utterances from 15 dysarthric speakers, each assessed by five naive listeners whose transcription accuracy assigned every speaker to one of four intelligibility classes: Very Low (0–25 percent), Low (26–50 percent), Medium (51–75 percent) and High (76–100 percent). Because intelligibility labels are tied to dysarthria severity, these constraints measurably reduce how well classifiers generalize to unseen speakers.

The new approach attacks the problem from a generative direction. The team filtered the TORGO recordings, discarding non-target utterances such as experimenter instructions and clips shorter than 0.4 or longer than 60 seconds, leaving 5,553 audio clips from 1,119 excluded items. Grad-TTS was then trained on those clips with the eight dysarthric speakers’ identities encoded as conditioning inputs, so that at inference the model could pronounce any text in a given speaker’s pathological voice. Feeding it the UA-Speech vocabulary produced synthetic speech designed to replicate TORGO speakers’ articulatory and phonatory traits over entirely new lexical material — a cross-corpus transfer that the authors say no prior study had investigated for intelligibility classification.

For evaluation, the researchers trained ResNet-50 and ResNet-152 classifiers on Mel spectrograms under a speaker-independent three-fold cross-validation scheme on the 15 UA-Speech dysarthric speakers. In each fold, four speakers — one per intelligibility class — were held out for testing, and no utterance of a test speaker was used at any stage of training. Four training conditions were compared at augmentation ratios of 33, 66 and 100 percent: an unaugmented real-data baseline, conventional signal-level augmentation with SpecAugment (pitch shifts of ±0.5–2 semitones plus frequency and time masking), the Grad-TTS synthetic corpus, and a Mix condition splitting the fixed augmentation budget equally between the two. Crucially, all three strategies added the same number of samples with identical class distributions, so any performance differences reflect the composition of the added data rather than its sheer volume.

The results favor the synthetic route in most settings. In the four-class task, the combined Mix strategy delivered the largest average gains: with ResNet-50, utterance-level accuracy rose from 0.302 to 0.377 and F1-score from 0.273 to 0.351, while speaker-level accuracy improved from 0.417 to 0.472. With ResNet-152, Mix achieved the best average speaker-level scores — accuracy 0.500 and F1-score 0.394, gains of 0.167 and 0.130 respectively, the largest single improvement observed in the four-class task. Per-class analysis showed the benefit concentrated on the intermediate Low and Medium classes, precisely those for which the real corpus provides the fewest speakers, while the Very Low class, which received no synthetic support because no TORGO speaker maps to it, actually degraded with ResNet-50 in all augmented conditions.

The binary task, collapsing the four classes into severe and mild groups, told a more nuanced story. With ResNet-50, all three strategies beat the baseline of 0.659 utterance-level accuracy, and Mix reached 0.705 accuracy and 0.700 F1-score — the only setting where conventional augmentation outperformed generative augmentation at the utterance level, possibly because a coarser binary decision depends less on fine-grained pathological detail. At the speaker level, however, the ordering reverted: Grad-TTS reached 0.805 accuracy against SpecAugment’s 0.778, and Mix was strongest overall at 0.889 accuracy, equivalent to correctly rating an average of 3.6 out of 4 held-out patients per fold versus 3 out of 4 for the baseline. With ResNet-152, Grad-TTS alone was the best condition in every column, while SpecAugment produced no utterance-level gain and an outright speaker-level loss.

An independent quality check tempered the enthusiasm. The team ran two reference-free mean opinion score predictors, UTMOS and DistillMOS, over the real and synthetic corpora. Both predictors track intelligibility only partially in real dysarthric speech, and the synthetic corpus compressed the dynamic range severely: the span between its Low and High classes was about one-sixth of the corresponding span in TORGO for both predictors (UTMOS 0.12 versus 0.76; DistillMOS 0.06 versus 0.38). The authors caution that this compression is not evidence that pathological traits were preserved or lost — reduced naturalness scores are equally compatible with synthesis or vocoder artifacts — but it does mean the synthetic utterances are only weakly differentiated across intelligibility classes, so labels inherited from source speakers may not correspond to acoustically distinct material.

The authors are explicit about the limits. Differences between conditions are of the same order as cross-fold standard deviations; each fold contained only four test speakers, making speaker-level accuracy quantized in steps of 0.25; the synthetic corpus is conditioned on just eight source speakers and covers a narrower perceptual range than the real corpora; and the mapping of TORGO speakers to UA-Speech classes relied on mean Frenchay Dysarthria Assessment subscale scores, a related but non-equivalent construct that may inject label noise. Early stopping and model selection also relied on validation utterances from speakers seen during training. The optimal augmentation ratio varied across architectures and tasks, suggesting it should be treated as a tunable hyperparameter rather than a fixed recommendation.

Within those limits, the study makes a pointed contribution: it shows that diffusion-generated speech from a small set of real patients can add a form of clinical variability that signal-level perturbations cannot, and that the two strategies are complementary in a specific sense — generative augmentation aims to simulate additional patients and mitigate speaker scarcity, while conventional augmentation regularizes the classifier against overfitting to comparatively homogeneous synthetic material. The team plans to explore ordinal classification approaches that explicitly model the natural ordering of intelligibility levels, and to use phonemic and prosodic analysis of per-class errors to identify the acoustic features driving confusion between adjacent classes. The authors also stress that the binary classifier should not be used as an autonomous screening instrument; at most it can serve as a decision-support signal alongside clinical judgement.

Subject of Research: Cross-corpus generative AI data augmentation for dysarthric speech intelligibility classification

Article Title: Improving dysarthric speech classification with generative AI-based data augmentation

Article References: Lillini, D., Basili, T., Migliorelli, L., Aironi, C., Gabrielli, L., & Squartini, S. (2026). Improving dysarthric speech classification with generative AI-based data augmentation. Medical & Biological Engineering & Computing. https://doi.org/10.1007/s11517-026-03705-x

Image Credits: AI Generated

DOI: 10.1007/s11517-026-03705-x

Keywords: dysarthria, speech intelligibility, Grad-TTS, diffusion model, data augmentation, TORGO, UA-Speech, ResNet, deep learning, speech synthesis, SpecAugment, cross-validation