AI speech model classifies dysarthria severity in real-world recordings

A proof-of-concept artificial intelligence system called Whisper-ST1 achieved 79% per-speaker F1 when classifying dysarthria severity from real-world audio samples. The system also showed strong agreement with ratings from two speech therapists, according to the study. The results support the feasibility of using a pretrained speech-recognition model to assess motor speech impairment under less controlled audio conditions.

Dysarthria is a motor speech disorder affecting people with neurological conditions and is highly prevalent among stroke survivors. Effective treatment requires assessment of severity and speech exercises tailored to the degree of impairment. Traditional speech intervention can face limited access to therapists and low treatment adherence. Whisper-ST1 was developed as a possible component for home-based rehabilitation, with the study focusing on automated severity classification rather than evaluating a complete rehabilitation program.

The researchers built Whisper-ST1 by fine-tuning a pretrained automatic speech recognition model using real-world data alongside traditional research datasets. The model architecture was adapted for the new task, and the researchers used progressive unfreezing to promote flexible adaptation while limiting catastrophic forgetting of relevant speech features. They also applied an ordinal regression loss to reduce the impact of misclassification errors. Ablation experiments examined the contribution of individual pipeline components in isolation.

On real-world samples, Whisper-ST1 produced a per-speaker F1 score of 79% [67%–100%] for dysarthria-severity classification. Its weighted Cohen’s kappa values were 0.814 [0.400–1.000] and 0.878 [0.647–1.000] when compared with the two speech therapists’ ratings, while inter-therapist agreement was 0.851 [0.550–1.000]. Model errors were predominantly between adjacent severity levels, suggesting that the system preserved the ordinal structure of the severity scale. The study therefore demonstrates a proof-of-concept classification approach in heterogeneous audio conditions and provides a foundational step toward AI-assisted tools for remote monitoring and support of dysarthria rehabilitation, but it does not establish the system as a replacement for professional assessment or a complete home rehabilitation intervention.

Subject of Research: Proof-of-concept evaluation of an AI-based audio classification system for dysarthria severity assessment

Article Title: An artificial intelligence-based audio recognition system for dysarthria home-based rehabilitation: A proof-of-concept study

Article References: Sancenon, V., Oh, P. Y. S., Tay, E. K. W., Lee, J. Z. X., Soon, M. L. E., Goh, H. L., Oey, N., & Ta, A. W. A. (2026). An artificial intelligence-based audio recognition system for dysarthria home-based rehabilitation: A proof-of-concept study. PLOS Digital Health, 5(10), e0001794. https://doi.org/10.1371/journal.pdig.0001794

Image Credits: AI Generated

DOI: 10.1371/journal.pdig.0001794

Keywords: dysarthria, artificial intelligence, speech recognition, motor speech disorder, Whisper-ST1, stroke, remote monitoring, ordinal classification, clinical speech assessment, home rehabilitation, artificial, intelligence-based