Exploring the Role of Machine Learning in Diagnosing and Treating Speech Disorders: A Systematic Literature Review

Introduction

Humans are inherently social creatures, with an innate inclination towards engagement and interaction. In this context, speech as verbal messaging is a unique characteristic of humans, and it plays a leading role in humans’ capacity to convey their thoughts, concerns, and perspectives to others¹. However, individuals with speech impairments encounter significant academic, psychological, and social challenges while engaging with their communities.^2–4 The number of individuals with disabilities is continuously increasing. The World Health Organization states that about 1.3 billion people with a disability worldwide need assistive technology (AT)⁵. This number could increase by 2030 to about 2 billion people. The UN Convention on the Rights of Persons with Disabilities (UNCRPD) has confirmed AT provision as a fundamental human right.⁶

Many interventions in the context of speech disorders are detected. Based on the causes of underlying speech disorders, some studies have provided treatment or assistance interventions for individuals with speech impairments, such as.^7–10 While others apply machine learning and deep learning methods to detect, classify, predict, and assess speech disorders; among them are^11–15 Machine Learning (ML) is a dominant branch of artificial intelligence (AI), covering remarkable advancements in research and industry. Machine learning showed notable impacts on improving communication tools for individuals with speech impairments as they enhance the accuracy and accessibility of speech recognition and word predictability, such as AI-driven speech-to-text and text-to-speech applications. Moreover, ML provides a host of powerful, automated algorithms designed to handle vast amounts of data across various disciplines like speech recognition^16,17 Natural Language Processing^18,19 human-computer interaction,²⁰ computer vision²¹ health informatics,²² recommender systems,²³ vocabulary context-aware prediction²⁴ and more.

Recent research demonstrated that deep signal analysis of voice using ML techniques to recognize speech with disorders showed promising results by extracting significant features from these signals, such as Mel-frequency Cepstral Coefficients (MFCCs) and Spectro Temporal utterances. Combining these two features shows more reliable results than others.^25–27

Notably applying ML techniques in speech recognition and augmented communication, enhancing accessibility and user experience, predictive and contextual communication, voice synthesis, and personalized language models.^25,28 ML models can learn and adapt from ML-powered ATS users. Collecting, annotating, and analyzing large and diverse datasets of disordered speech samples would enable ML algorithms to identify specific users’ speech patterns and nuances. That helps develop personalized models integrated into assistive devices, such as speech-generating or voice-recognition systems.^29,30 Moreover, ML approaches as data-driven approaches can play a valuable role in diagnosing and treating speech disorders.³¹

Despite the advantages of the SLR mentioned above, it is imperative to acknowledge the subsequent drawbacks. Most proposed SLRs focused on only one type of speech disorder, such as,^32,33 where only aphasia and Dysarthria are studied, respectively. Other SLRs^34,35 pay attention to one patient’s age: children’s age. In,³⁶ the focus is on assistive technologies used.

The present systematic literature review aimed to identify, categorize, and compare effective speech disorder detection for analyzing multiple speech disorders suitable for all age categories instead of choosing only a particular disorder or speech analysis tool as observed in the existing reviews. The proposed inclusive systematic review seeks to study the role of ML approaches in identifying, classifying, and evaluating these disorders. In addition, the study focuses on the ML role in treating these disorders, considering their potential causes, either biological, psychological, or environmental, regardless of the presence of cognitive impairments such as Down’s syndrome³⁷ or Alzheimer’s disease.^38,39 This work aims to comprehensively analyze the ML techniques for speech impairment recognition, focusing on the challenges and limitations. The primary contribution of our work implies:

• Providing a review highlighting the existing ML methods, algorithms, features extraction techniques, models’ performance metrics, and the characteristics of the obtained datasets, focusing on discovering the state-of-The-art of all these techniques utilized by scholars in the field.
• Identifying the existing categories of speech disorders and clarifying how different ML approaches address these disorders.
• Shedding light on the limitations and challenges in the existing ML-based speech disorder detection, classification, and evaluation.
• Identifying gaps and potential opportunities for further research and improvements.

To achieve the review aims, we conducted the present systematic literature review following the guidelines outlined in the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) statement for a systematic review.⁴⁰ This protocol allowed us to carefully select the related studies, extract pertinent information, and present the results focusing on addressing the research questions. The rest of this paper is organized as follows: Background presents the background, containing the essential concepts related to our study. Research Methodology describes the methodology used to collect and select the articles studied and the research question. Discussion analyzes existing ML solutions for speech disorder recognition by answering the research questions. Section 5 presents new research directions to improve assistive technology for speech disorder users. In the final section, we conclude the paper by summarizing the work and highlighting future directions.

Background

This section will introduce the significant concepts pertinent to our research.

Speech Disorder

Speech is the central procedure of communicating thoughts, emotions, and ideas to others. It involves the sophisticated coordination of various body parts, such as the head, neck, and chest. This coordination is necessary for effective interaction. A speech disorder is a health condition that impairs a person’s capability to utter words due to damage to muscles, nerves, or vocal structure. Speech disorders are complicated and varied conditions that can be shown in several forms, such as stuttering, Dysarthria, aphasia, Parkinson’s Disease, Apraxia of speech, stammering, phonological disorders, and ataxia.

In the broader literature, the term “speech and language disorders” is categorized under communication disorders disability, alongside hearing disorders, deafness, and physical disabilities that impact speech, as depicted in Figure 1 ⁴¹ which presents the most known speech disorder:

• Dysarthria: considered as a physical disorder. According to the American Speech-Language-Hearing Association (ASHA),⁴² Dysarthria is a motor speech disorder caused by muscle problems. It can make it hard to talk. As a sign of dysarthria, we can find speech that is too soft or too loud, the sound that is hoarse or breathy, etc.
• Aphasia is a type of language disorder affecting the ability to make or understand speech and read or write, as defined by the National Aphasia Association,⁴³ The cause of aphasia is always a brain injury, most often from a stroke, especially in older people.
• Dysophania: Dysphonia International describes Dysphonia as occurring when there is a change in the normal vocal tone, which could result from a structural or functional issue.⁴⁴ It is characterized by altered vocal quality, pitch, loudness, or vocal effort (shaky voice; rhythmic pitch and loudness undulations).
• Parkinson’s Disease: Parkinson’s Foundation defines Parkinson’s Disease as: “A neurodegenerative disorder that affects predominately the dopamine-producing (‘dopaminergic’) neurons in a specific area of the brain called substantia nigra”. Parkinson’s disease is characterized by hypokinetic dysarthria, which is featured by abnormalities like the inability to maintain loudness, monotonous and harsh voice, articulation errors, and reduced fluency.^45,46
• Apraxia of speech: Apraxia is a neurological disorder in which people cannot do learned movements on command, even though they know what they are supposed to do and are willing to do it.^2,47 A patient with Apraxia of speech has difficulty moving their mouth in the way needed to produce sounds and words.
• Stammering/Stuttering: As defined by the British Stammering Association, it is a speech disorder that involves frequent and significant problems with normal fluency and flow of speech.⁴⁸ A symptom of stammering is a person can repeat sounds or words, stretch or prolong sound (eg “Hello fffffffreind”), and there is a silenced spot where a sound gets stuck.
• Phonological disorders: Also called speech-sound, it happens when people have trouble making certain sounds, even if there is no physical reason for the problem. Lisp is an example of this type of speech disorder. As an example of phonological disorder signs, the patient leaves off sounds from words, like saying “coo” instead of “school”.

Figure 1 Speech disorder taxonomy. Note: Adapted from Defining Speech and Language Disorders; 2023. Available from: https://speechandlanguagedisabilities.weebly.com/. Accessed December 11, 2023.41

These disorders can substantially affect a person’s transmission abilities and overall superiority of life. Seeking qualified help and treatment alternatives is crucial for handling and improving speech disorders. Unfortunately, the number of individuals with speech disorders continuously increased, as declared by the World Health Organization. Table 1 shows statistics on speech and language disorders around the world.

Table 1 Statistics of Speech and Language Disorders

Machine Learning

Machine Learning (ML) models have emerged as valuable tools in speech disorders that have significantly empowered people with these disabilities through cutting-edge assistive technology solutions. AI and ML can analyze big data, identify patterns, make predictions, and imitate human cognitive functions.⁵⁴

Machine learning, often abbreviated as ML, is the subfield of Artificial intelligence that intends to enable computers to learn from data and make predictions without being explicitly programmed. It is gaining more and more attention due to its significant role in many fields, including healthcare, manufacturing, finance, speech disorders, and more. It powers many technological advancements, like speech recognition, recommendation systems, self-driving cars, and predictive analytics. The main goal of machine learning is to build a model that performs well on both the training and test datasets. Data, comprising features and labels, is used for model training. During training, the model learns patterns and relationships between features and labels. The trained model is then tested on a separate dataset and used for inference. Machine learning algorithms can be classified into several categories, as illustrated in Figure 2:

• Supervised Learning: In this type of learning, the algorithm is trained on a labelled dataset denoted as (X,y), where X represents the input features, and y represents the corresponding output labels. In supervised learning, the primary objective is to learn how to make predictions or classifications based on this labelled dataset. The regression and classification techniques are the primary techniques in this category.
• Unsupervised Learning: In this setting, the algorithms are trained on unlabeled datasets X. It aims to find patterns, groups, and structures within the datasets. Clustering and dimensionality reduction are commonly utilized techniques in this category.
• Other variations: This type of ML includes, but is not limited to, semi-supervised Learning and Reinforcement Learning. For instance, semi-supervised Learning is a hybrid method that combines supervised and unsupervised learning. Artificial neural networks are the most used in speech recognition.

Figure 2 Machine learning taxonomy.

Automatic Speech Recognition (ASR) for Speech Disorder

The main component of assistive technologies for people with speech disorder disabilities is Automatic speech recognition, which is the process by which a computer can recognize and act upon spoken language or utterance.¹² An ASR, as illustrated in Figure 3, can produce a text from a speech by analyzing and processing speech signals using different ML techniques, such as Convolutional neural networks²⁹ or Deep Learning.³⁰ Indeed, the primary objective of the ASR system is to evaluate the various speech signals concerning various phonemes, syllables, and words/sentences. In the context of speech disorder, the patient can use an ASR to detect voice disorder and the voice pathologist to make an intelligent assessment of the patient.⁵⁵

Figure 3 ASR Architecture.

The performance of the ASR mainly depends on the training dataset, which is sorted into training and testing sets by randomly selecting the observations from healthy and sick voices.⁵⁶ The learning set is used to build the machine learning model, while the testing set is used to evaluate the final model’s performance and generalization.³⁵ We need to process and turn the user’s speech into a set of features to use ML algorithms.

Before extracting the features, the original speech signal has to go through preprocessing, which is the initial and most crucial step in the automatic speech recognition process. This step consists of cleaning the speech signal from ambient and undesirable noises, detecting speech activity, and normalizing the length of the vocal tract. The purpose of preprocessing a speech signal is to enhance the computational efficiency of speech recognition systems⁵⁷ by utilizing various preprocessing techniques, such as speech pre-emphasis, vocal tract length normalization, voice activity detection, noise removal, framing, and windowing.

The feature extraction procedure involves identifying the audio signal components that can be used to identify linguistic content while removing background noise and irrelevant information. In general, feature extraction is the process of generating the speech signal in digital form. Features can be mainly categorized into four categories: linguistic, contextual, acoustic, and hybrid. Various feature extraction techniques can be used in this first step, such as:

• Acoustic analysis measures the sound information in a speech to extract features related to phonation, articulation, Prosody, voice quality, etc.¹³ For instance, articulation features can be vowel quality, coordination of laryngeal and supralaryngeal activity, precision of consonant articulation, tongue movement, occlusion weakening, and speech timing. Prosodic features can be pitch, loudness, and duration⁵⁸ In contrast, voice quality involves jitter, shimmer, first three formants, and harmonic-to-noise ratio.
• Mel-frequency cepstral coefficient (MFCC): used to represent the audio signal power spectrum and to record the timbral information of sounds.^59,60 The MFCCs are a set of coefficients that together form a Melfrequency cepstrum. MFCCs provide a suitable number of frequency channels to analyze audio, with only 12 parameters related to the amplitude of frequencies.
• Glottal Flow Signal: The glottal flow refers to the airflow that originates from the lungs and proceeds through the vocal folds in the larynx. The vocal folds vibrate, causing them to open and close periodically. An inverse filtering of the voice signal can obtain the glottal flow signal. Many parameters can be obtained from the glottal flow signal, but they are unsuitable for speech disorders.⁶¹ Time-domain features are made by measuring how strong the speech signal gets over time. Time-domain features include energy, zero-crossing rate, pitch, and Linear predictive coding (LPC). Frequency-domain parameters where features are made up of the signals’ frequency domain, also called its spectrum.⁶²
• Spectro-temporal sparsity is mainly related to the diversity of disordered speech.⁶³ The main goal of the spectral features is to learn characteristics such as volume reduction, changes in format position, imprecise articulation, and hoarse voice. At the same time, the temporal features aim to capture patterns such as increased disfluencies and pauses.
• Discrete Wavelet Transform (DWT): Aiming to analyze non-stationary signals with multi-resolution potential, the Wavelet transform can be used as a time-frequency transform. DWT can do both the pathological voices’ time and frequency domain analyses.⁶⁴ Thus, it is incredibly useful for detecting vocal issues.

One of the critical challenges in any ASR system is the number of features that can increase the cost of computation time and the system’s performance. As a solution, feature selection can reduce the number of features by removing redundant and irrelevant features and boosting system performance. Many techniques of feature selection exist, such as Support Vector Machine-Recursive Feature Elimination (SVM-RFE)⁶⁵, minimum Redundancy Maximum Relevance (mRMR),⁶⁶ Chi-square,⁶⁷ and Principal Component Analysis (PCA), Local Learning-Based Feature Selection (LLBFS),⁶⁸ and Least Absolute Shrinkage and Selection Operator (LASSO).⁶⁹ For instance, LASSO modifies the absolute value of feature coefficients; a feature with a coefficient that becomes zero will be removed from the set of features. Authors⁷⁰ use LASSO, LLBFS, Relief, and mRMR feature selection methods.

Related Work

Although many works have been proposed in the literature,^{11–15,71,72} to our knowledge, few reviews have explored using AI and ML in identifying, predicting, and assessing different speech disorders. For instance,³² surveyed existing works on automatic assessment systems designed to evaluate patients’ aphasia and the severity level of patients. In another study,³³ a review focused on the characteristics of dysarthric speech and introduced assistive solutions like robust automatic speech recognition (ASR) systems. Meanwhile,⁷³ another study comprehensively analyzed the different voice disorders. They explored the existing machine learning (ML) approaches leveraged to develop automatic detection systems for voice disorders.

Additionally,³⁶ a systematic review delved into applying various ML models on the Internet of Things (IoT)-based Assistive Technology research. The study focused on the context of these models’ applications and examined the IoT devices that cater to people’s cognitive, hearing, visual, and degenerative diseases. In another systematic review,³⁴ studies involved in speech assessment methods for children and adolescents with different speech impairments were presented. A few ML-based approaches are presented in this study.

A systematic literature review of online speech therapy systems for childhood speech communication disorders is presented.³¹ To compare these systems, authors used the following criteria: features of the proposed system, end user, used ML algorithm or not, and evaluation metrics. A SLR dedicated to an automatic Speech Recognition System for Tonal Languages is proposed in.⁷⁴

Despite the advantages of the SLR, as mentioned above, it is imperative to acknowledge the subsequent drawbacks. It is clear to notice that the majority of proposed SLR focused on only one type of speech disorder, such as³² and,³³ where only aphasia and Dysarthria are studied, or on a specific type of language, such as Tonal Languages in.⁷⁴ Other SLRs^34,35 pay attention to one patient’s age. In,³⁶ the focus is on assistive technologies used as assessment technology for speech disorder patients.

Compared to the SLR, as mentioned earlier, the present systematic literature review aimed to identify, categorize, and compare effective speech assessment methods for analyzing multiple speech disorders suitable for all age categories with speech disorder disability instead of choosing only a particular disorder or speech analysis tool as observed in the existing reviews.

Research Methodology

This section outlines the methodology that was employed to conduct the subsequent investigation. Our research strategy was conducted in three phases. The first and second phases consisted of article selection processes by defining the paper selection strategy and research questions, and the third phase was data synthesis, as presented in the discussion section.

Research Strategy

This study will discuss and present a detailed exploration of machine learning (ML) and deep learning (DL) techniques in categorizing, recognizing, and predicting speech disorders. The study seeks to provide a detailed overview of the progressions in this discipline, shedding light on the various engaged feature extraction techniques, algorithms, datasets, and methodologies.

Based on our research questions, we defined principal keywords to research existing approaches in the literature dealing with this topic and obtained results from various databases. After gathering all the articles from the sources, we applied our filtering rules. First, we kept only recently published papers from the last ten years. Then, we filtered the results based on title, abstract, and Keywords. A second filter is applied to keep only papers published in reputable journals or international conferences classified as A or B. The overall process is shown in Figure 4.

Figure 4 PRISMA protocol-based paper selection process.

Search Databases Selection

The databases used for our search are ACM,⁷⁵ IEEE,⁷⁵ ScienceDirect,⁷⁶ Springer,⁷⁷ and Taylor & Francis. Papers were selected based on their titles, abstracts, and keywords. We considered journal papers ranking Q1 or Q2 and conference papers of classes A or B. The number of papers obtained from each database at each step of our selection process according to the PRISMA protocol is discussed in the coming sections. We finally obtained 65 papers that were highly relevant from the different databases.

Search Keywords

After the final step of our paper selection process, we proceeded to full-text screening, which gave us the exact theme related to this survey. After filtering by title, abstract, and keywords, a final set of 65 papers was chosen to conduct this survey based on their relevant content. We used an automatic search method and used the following search strings using keywords set from Table 2: one instance of “keyword set 1” AND one instance of “keyword set 2”. We also used combinations of search strings from the different lines of the keywords table using the “OR” operator. For example: “Speech disorders” OR “Assistive speech disorders” AND “ML”.

Table 2 Search Keywords Definition

Inclusive/Exclusive Criteria

The inclusion and exclusion criteria determine the systematic literature review’s scope. They are first defined after deciding on the research issue and before accomplishing the search, but scoping searches could be necessary to choose pertinent inclusion and exclusion criteria. Various rules may be used to define these criteria, as shown in Table 3. A set of 65 papers was obtained after applying the criteria in Table 3.

Table 3 the Inclusive and Exclusive Search Criteria

Research Questions

This study will present a detailed exploration of the function of machine learning (ML) techniques in addressing speech disorders. The study seeks to provide a detailed overview of the progressions in this discipline, shedding light on the various engaged feature extraction methods, preprocessing techniques, datasets, and performance metrics. The underlying empirical question of this review is: What are the current ML algorithms? The study investigates the extent to which the models are comprehensive and inclusive for detecting and classifying speech disorders. All the scientific studies are synthesized to provide evidence for the following specific questions:

Q1: What details of the bibliographic profile are within the realm of existing studies?

Q1.1: What types of speech disorders are included in the existing studies?

Q1.2: How has the number of studies on this topic changed over the years?

Q1.3: What platforms (eg, journals, conferences, workshops) were selected by studies authors for dissemination?

Q2: What datasets and languages were used in the studies?

Q3: What preprocessing procedures are employed in constructing machine learning models?

Q4: What feature extraction and classification are prevalent in the studies?

Q5: What are the existing ML algorithms for speech disorder recognition?

Q6: What performance metrics have been used to gauge the efficacy of the proposed ML models?

These questions aim to thoroughly explore the ML field in speech disorders, focusing on various aspects like types of disorders studied, the evolution of the research over time, methodologies used, and the effectiveness of different approaches.

Discussion

In this section, we synthesize the analysis of the research papers proposing an ML-based solution for patients with speech disorders and provide the answers to the identified research questions. In total, the selected papers discussed in this paper are solutions.

Q1: What details of the bibliographic profile are within the realm of existing studies?

Q1.1: What types of speech disorders are included in the existing studies?

Of all the papers studied, we have distinguished several types of speech impairment problems: impaired vowel articulation, Dysarthria, Aphasia, Dysphonia, Apraxia of speech, stuttering, stammer, and Phonological disorders. We could classify these problems according to the number of papers dealing with them. Figure 5 summarizes the papers dealing with the same problem. We can conclude that most of the papers have focused on the problem of Dysarthia, as it is the most common speech impairment disorder. Some papers in this category were only focused on dysarthric people with Parkinson’s disease. The other speech impairments were treated with fewer papers like Aphasia, Apraxia, Dysphonia, and Dysphagia. We classified the rest of the papers under “Speech impairment” as dealing with specific problems like imprecise vowel articulation or severe speech impairment problems.

Figure 5 Number of papers per speech impairment type.

Q1.2: How has the number of studies on this topic changed over the years?

According to Figure 6, we can conclude that the problem of speech disorders has recently gained increasing attention from researchers due to the significant technological progression in the development of automatic speech detection systems (ASR). Approximately a quarter of the papers were published in the most recent year, with a noticeable percentage in the previous two years. ASR systems have extended the horizons for new methods in dealing with persons with different speech impairments.

Figure 6 Classification of studied papers by year.

Q1.3: What platforms (eg, journals, conferences, workshops) were selected by the studies’ authors for dissemination?

We analyzed the publications’ source by year and publication source, as shown in Table 4. According to this table, we can conclude that most of the publications were made at ACM, which publishes a well-known journal related to this issue, namely the “IEEE/ACM Transactions on Audio, Speech, and Language Processing”.

Table 4 Showing the Classification of Studied Papers by Year and Publication Source

Q2: What datasets and languages were used in the studies?

This section will provide insight into the datasets used in the literature, significantly impacting the investigation’s precision and progress. In studied papers for speech disorder impairments, all datasets are mainly real-world datasets built from patients and healthy speakers. We can categorize these datasets into two categories: public and private. Public datasets are offered at publicly available sources such as TOROGO,⁷⁸ and UASpeech.⁷⁹ The second category contains datasets that, to be used, we need to contact authors such as,^80,81 and.⁸² Table 5 shows a comparison of datasets used by papers. This comparison is mainly based on features of each dataset, such as speech disorders type, languages supported, and instance.

Table 5 Summary of Datasets Used by Retained Papers. M = Male, and F= Female

From Table 5, we can derive the subsequent observations:

• As we have examined various databases and classified them based on the languages of each database, it is evident that English is the predominant language, constituting over half of the data, followed by French at around 20%. Other languages like Spanish, Japanese, Italian, and Korean are represented to a lesser extent.
• Public datasets, including TOROGO and UASpeech, are frequently used in multiple research papers.
• The datasets primarily consist of recorded sentences from speakers with a balanced gender distribution. These speakers are typically divided into roughly equal groups of patients and healthy individuals, although there are exceptions in some datasets, such as ATR,¹¹³ TIMIT,¹²⁹ and EMA.¹³²
• Not all datasets are exclusively focused on dysarthria patients; some, like the EMA dataset,¹³² are applicable in other areas.
• Regrettably, the Arabic language receives less attention, suggesting a scarcity of studies targeting Arabic-speaking individuals with speech disorders.

Q3, what preprocessing procedures are employed in constructing machine learning models?

Across the retained studies, researchers have used different pre-processing steps depending on the used dataset and the features they intend to employ. In general, the most used pre-processing techniques include:

• Normalization: This process involves normalizing audio signals to standard amplitude levels to ensure consistency and improve system durability. Methods used for normalization may involve scaling the signal within a specific range, such as between −1 and 1,^70,105 or employing z-score¹³¹ normalization or techniques like peak normalization.
• Noise Removal / Filtering: Since Automatic Speech Recognition (ASR) systems are sensitive to ambient noise, adversely affecting recognition accuracy, noise reduction techniques are crucial. These techniques, including spectral subtraction or adaptive filtering, are applied to reduce the impact of ambient noise.
• Signal Segmentation: It allows the continuous audio stream to be broken into smaller segments, often based on pauses or other criteria, helps handle long audio recordings, and aligns the speech with linguistic units during recognition.
• Data Augmentation: It helps increase the training data’s diversity and improve the model’s robustness. Commonly applied augmentation methods include velocity perturbation, pitch shifting, time stretching/compression, noise injection, jitter, dynamic range compression, and room impulse response to simulate real-world conditions. Considering specific speech characteristics associated with Dysarthria, such as pitch, rate, and quality changes, these techniques help create a diverse and more representative training dataset, such as in.¹⁰⁰
• Down-sampling recordings: This refers to the process of reducing the sampling rate of a recording. The sample rate represents the number of samples taken per second to represent a continuous audio signal digitally. Down sampling can benefit computational efficiency and resource usage by reducing the amount of information that needs to be processed, which can benefit training and inference speed, especially when working with large data sets, as in.^63,85,105
• Signal alignment refers to synchronizing or aligning two or more signals in time. In ASR, signal alignment is often used to align the input speech signal with a reference or sample. This alignment ensures that the corresponding features or segments in the two signals match exactly, facilitating identification or comparison. Dynamic Time Warping (DTW) is a common technique to align signals.
• Signal transformation involves converting signals from one representation to another, which allows for extracting meaningful information or preparing data for analysis. Common transformations include the Fourier transform, which represents the frequency components of a signal; the Wavelet transform, suitable for analyzing signals with non-stationary characteristics; and the Mel Frequency Cepstral Coefficient (MFCC), widely used in speech processing.

In the context of speech impairment, we chose the following criteria to compare preprocessing features mentioned above:

• Effectiveness in capturing impairment characteristics: The technique can be effective in capturing speech impairment characteristics, but it usually has limited results in the literature. Effectiveness measurement is more detailed for studied references in section 5.6 using metrics like accuracy and error rates.
• Preservation of the input information (or signal): indicates whether the specified technique preserves the input signal entirely or partially by deforming a part.
• Computational Complexity: it depends on the used algorithms, but we indicate here the degree of complexity of the algorithms usually used for each technique as follows: linear (for O(n)), quadratic (for O(nn)) and quadratic (for O(nlog n)).

Table 6 shows the comparison of the preprocessing techniques and methods that were applied in the reviewed papers. Signal segmentation and alignment techniques were the most used, with 55% of the reviewed papers using them. Signal alignment, primarily through techniques such as Dynamic Time Warping (DTW), is essential in automatic speech recognition (ASR) for speech disorders because it corrects temporal irregularities and variations in speaking rate, allowing accurate comparison with reference signals. Signal transformation, such as using Mel-Frequency Cepstral Coefficients (MFCC) in,^63,85,105 is also essential to create informative feature representations that capture unique features of brain disorders. These preprocessing techniques improve the adaptability and robustness of ASR systems, allowing them to effectively recognize speech patterns affected by different types of impairments.

Table 6 Comparison of Commonly Utilized Preprocessing Methods

Normalization, down-sampling, and noise reduction are advantageous for treating speech impairment in ASR systems as they allow standardizing the amplitude of speech signals and ensuring their consistency by mitigating variations in loudness, contributing to a more uniform dataset. However, data augmentation is a crucial technique in this issue, typically in scenarios where the available data may be limited. Augmentation strategies can be tailored to reflect specific challenges posed by different impairments, making the model more robust and adaptable

Q4. What Feature Extraction Techniques are Prevalent in the Studies?

Feature extraction is crucial in automatic speech recognition (ASR) for speech disorders. This involves converting the raw audio signal into representative features that capture the information needed for recognition. Multiple feature extraction techniques are commonly used, emphasizing robust representation of speech disorders. Table 7 displays commonly employed methods in the studied works, the proportion of their utilization, and the research studies that relied on these methods.

Table 7 The Commonly Utilized Extraction Methods. In Some Cases, Studies Incorporate More Than One Technique. Following That, the Same Study Was Replicated, Thereby Increasing the Overall Number of Research Studies

More particularly, Mel frequency cepstral coefficients (MFCC) are frequently used in automatic speech recognition (ASR) systems to detect speech disorders due to their effectiveness in capturing essential characteristics of the signal voice, especially when something goes wrong. MFCCs mimic the sensitivity of the human auditory system to different frequencies, making them robust to variations in spectral characteristics caused by speech disorders. The ability to represent the power spectrum of speech signals in a compact and discriminatory manner makes MFCC well-suited for recognizing patterns associated with speech disorders. Additionally, MFCCs provide a good balance between capturing relevant information and reducing dimensionality, and thus, they are computationally efficient for use in ASR systems to detect speech disorders.

Table 7 shows that MFCCs-based features and Spectro-Temporal-based features constitute the most often utilized features among the researchers derived from the retained studies. The MFCC approach was used in 23 of 65 retained studies or 35.4% of the examined publications. In addition, we noted that the researchers relied heavily on spectrogram analysis in extracting the acoustic features of speech signals. Although the researchers found several ways and techniques for better detecting Dysarthria, analysis indicates that the combination of MFCC and Spectro Temporal utterance methods, such as,^{87,94,124,138} achieved better accuracy than others.

Detecting and investigating how the patients articulated utterances or words has attracted the researcher, and we have noted many studies (18.5% of the reviewed studies). Articulation investigation involves assessing the accuracy and clarity of speech, which often occurs in speech disorders. Articulation evaluation techniques often involve analysis of formant frequencies, articulatory alignment patterns, and phonetic features extracted from speech signals. Speech timing, on the other hand, focuses on the temporal aspects of speech production and examines changes in speech rate and rhythm. Temporal features such as pause duration and speech rate are often used for speech timing analysis. The main advantage of articulatory assessment techniques is their ability to identify specific tonal distortions and inaccuracies associated with voice disorders.

Regarding speech timing, these techniques provide insight into irregularities in the temporal structure of speech that may indicate and characterize specific disorders. Combining these techniques in ASR improves the diagnostic potential of voice disorders and provides a more nuanced understanding of both articulatory accuracy and temporal dynamics in voice disorders.

Finally, other studies were based on alternate speech feature extraction methods, such as analysis of electromyography/MRI images, occlusion weakening and word features, and lexical diversity. The analysis of electromyography (EMG) and magnetic resonance imaging (MRI) images provides valuable insight into the physiological aspects of speech production and can help detect speech disorders related to muscle activity or anatomical structures. Occlusal weakness analysis, focusing on speech patterns in cases of partial impairment or weakness, can help identify Dysarthria. Integrating word features and measures of lexical diversity can reveal patterns associated with linguistic challenges and vocabulary limitations, improving the diagnostic power of ASR in such issues. Moreover, combining several techniques helps provide a comprehensive approach for detecting language impairment in ASR by including physiological, articulatory and linguistic aspects for a more accurate and nuanced assessment.

Q5. What are the existing ML algorithms for speech disorder recognition?

As most ASR approaches rely on ML techniques, we found that 67% of the studied papers used machine learning (ML) methods in their speech recognition approaches. The rest of the papers present surveys or recognition tools dedicated to people with different speech impairments, and they used existing ASR systems from the literature. The overall distribution of ML algorithms with the corresponding preprocessing and feature extraction methods according to the studied references is given in Table 8.

Table 8 Classification of References per ML Algorithm Used, Preprocessing and Feature Extraction Methods

In Table 8, we have analyzed the different ML algorithms used. From this table, we can see that the most used algorithms are classifiers like SVM and LR. Support vector machines (SVMs) are widely used in automatic speech recognition (ASR) for speech disorders due to their effectiveness in classification tasks, especially in handling nonlinear decision boundaries. SVM efficiently handles high-dimensional feature spaces, making it suitable for complex acoustic features extracted from impaired speech signals. In the context of dysarthria detection, SVM provides a robust framework for capturing complex patterns in data, allowing better classification of dysarthric speech. SVM has become a popular choice in ASR systems for speech disorders because it handles nonlinear relationships between features and can adapt to different impairment characteristics, contributing to improved accuracy and generalization.

ASR systems for speech impairment using SVM use common preprocessing methods such as normalization, filtering, and down-sampling to ensure consistent and efficient feature representation. Several feature extraction methods could be used with SVM, such as MFCC, Spectral-temporal keywords, and Speech timing, allowing the capture of the speech’s relevant spectral and temporal characteristics. Because SVM operates in a high-dimensional space, it is adequate for classification tasks with complex patterns, such as distinguishing between normal and Dysarthric speech. This combination of preprocessing and feature extraction aims to create informative and discriminative feature vectors and optimize SVM performance for accurate speech recognition and fault detection.

For instance, in,⁷⁰ a machine learning system for diagnosing PD from speech signals is proposed to classify People with Parkinsonism from healthy ones. Six classifiers were used in this approach, ie, Adaboost, SVM, k-NN, MLP and NB. The experimental results indicated that SVM is the most successful classifier. In,¹³⁹ The goal is to detect PD patients by combining more than one symptom (rest tremor and voice degradation). Three classifiers were used: KNN, SVM, and NB. The majority vote technique is used to decide whether a person is PD. The proposed approach allowed them to achieve an accuracy of detection of PD up to 99.8% and confirmed that SVM classifiers take care of the outliers better than kNN and NB.

The approaches based on Artificial Neural Networks like CNN, DNN, GOP and MANN have been adopted to design a large set of ASR systems (33% of studied papers). Goodness Of Pronunciation (GOP) is a method based on both Convolutional Neural Network (CNN) and Deep Neural Network. Neural networks are widely valuable for ASR for speech disorders because they can automatically learn complex hierarchical representations from data, making them suitable for capturing complex patterns in impaired speech signals. More particularly, Convolutional neural networks (CNNs) effectively capture local patterns of spectro-temporal features, making them valuable for analyzing audio signals. Recurrent neural networks (RNNs) with sequential memory are good at modelling time dependencies and can help recognize subtle patterns in language disorders. Additionally, hybrid architectures such as GOP and MANN improve the alignment and decoding process.

With artificial neural network techniques, preprocessing often includes normalization, filtering, and down-sampling to ensure input consistency. Feature extraction methods usually used are MFCCs, and spectrogram plots provide helpful input to neural networks. The advantage lies in the ability of neural networks, especially deep learning architectures, to automatically extract hierarchical features and adapt to different characteristics of speech disorders. Also, the end-to-end learning approach minimizes the need for hand-crafted features and enables the model to recognize relevant patterns and nuances associated with different types of Dysarthria and speech impairment in general. The versatility and adaptability of neural networks make them powerful tools for ASR related to speech disorders.

For instance, inspired by the temporal processing mechanisms of the human auditory system, the authors of¹³¹ proposed a deep learning-based dysarthric speech detection technique that separately processes the temporal envelope and fine structure signals. Two discriminative representations learned from the temporal envelope and fine structure using CNNs are then exploited for automatic dysarthric speech detection.¹³¹ Other approaches combined the use of both traditional machine learning algorithms and ANN-based ones. In,¹³³ the authors combined the use of SVM and DNN to study the use of voice source information in detecting Parkinson’s disease (PD) from speech using traditional pipelines and end-to-end. The traditional pipeline used SVM classifiers to classify the speech utterances into healthy or PD labels based on the extracted features. In an end-to-end approach, they trained Deep learning models on raw speech waveforms and voice source waveforms using convolutional layers and multilayer perceptron (CNN and MANN). The experimental results indicated that the SVM-based approach achieves up to 67.93% accuracy, while the CNN approach achieves 68.56%. Another hybrid approach was proposed in,¹¹² where authors propose a hybrid framework in which generative models are used for learning representation and discriminative models are used for classification.¹¹² The proposed approach outperformed conventional HMM and DNN-HMM-based approaches for various intelligibility levels.

Moreover, a multi-networks speech recognizer (DM-NSR) model is proposed in⁸⁷ using a realization of the multi-views multi-learners approach called multi-nets artificial neural networks (MANN). In particular, the DM-NSR model employs several ANNs to approximate the likelihood of ASR vocabulary words and deal with the complexity of dysarthric speech.⁸⁷ Authors trained 443 neural networks. For the speaker-independent ASR system, the DM-NSR recorded an average recognition rate of 15.69% and a decreased error rate of 6.25%.

Q6. What performance metrics have been used to gauge the efficacy of the proposed ML models?

As most studied papers used ML or Deep Learning methods for their proposed ASR approaches, Figure 7 shows that Accuracy is the most used performance metric. For 15.7% of works, they used Pearson correlation coefficient (PCC)¹¹⁴ and Root mean square error (RMSE). Other evaluation metrics like sensitivity (which shows the ratio of correctly classified patients) and specificity (which indicates the percentage of correctly classified healthy people across the whole range of healthy people) were used.⁷⁰

Figure 7 Performance Metrics Employed in Machine Learning-Based Approaches.

Common ML metrics used in the studied papers are assessed as follows:

• Accuracy: It measures the model’s overall performance for all categories. It can be assessed using the following equation:

with TP being true positive and TN being true negative. FN is false negative, and FP is false positive.

• Recall: is the proportion of all true positives predicted by the model divided by the total number of predicted values.³² It can be evaluated using:

• Precision: calculates the proportion of correctly identified positives as follows:³²

• F1-score is a summary of both recall and precision and can be assessed as:⁵⁷

• PCC is a statistical method that calculates the correlation between two variables.¹¹⁴
  1. Root-mean-square deviation (RMSE) calculates the difference between the predicted values and the observed values as follows (eq. 5):¹³⁸

Moreover, speech recognition accuracy metrics play a predominant role in evaluating the performance of any speech-assistive communication aid.¹²⁷ For 12.28% of the studies, they used Error Rate metrics like Word Error Rate (WER) or Sentence Error Rate or SER and phoneme Error Rate (PER) or Accuracy Recognition Rates like Word Recognition Accuracy(WRA) or Sentence Recognition Accuracy (SRA). For instance, the Accuracy Recognition Rate in ASR systems is evaluated as the number of correctly predicted words, sentences, or phonemes by persons out of all the test databases. For example, WRA can be assessed using the following formula:¹²⁷

(6)

NC is the number of samples correctly recognized, and TC is the Total number of words per class. Word error rate (WER) is a common metric of the performance of a speech recognition or machine translation system. WER is the number of errors divided by the total words.⁸⁸

In Table 9, we present the values of achieved accuracy in the studied works grouped by the dataset used with other assessed metrics.

Table 9 Performance Metrics Values per Reference

We note from Table 9 that the larger the dataset size, the better the value of achieved accuracy, such as for the UA-Speech Database and the TORGO database.⁷⁸ The same observation is valid for large corpus specific to some languages, like in¹²⁰ using an Aphasia Bank corpus for 78 persons and in¹²⁴ using a specific Korean Dysarthric dataset for 174 persons. Some works were restricted to a specific language with limited datasets, like in,^12,107,123 but authors show that using techniques such as transfer learning could help generalize their approaches for multiple languages and achieve better performances, such as in.^14,84,94,128 Moreover, using standard datasets for Dysarthria persons helps compare experiment results with other works and better evaluate the techniques used.^26,87,98,112

In summary, in automatic speech recognition (ASR) systems for speech disorders, hybrid architectures such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), and connectionist temporal classification (CTC) have demonstrated superior performance in terms of achieved accuracy values. CNNs are effective for feature extraction because they are exceptionally efficient at capturing local patterns in spectro-temporal features. RNNs, particularly long short-term memory (LSTM) networks, can model temporal dependencies necessary for subtle speech patterns. Hybrid models like GOP combine the strengths of CNNs and RNNs to improve the alignment and decoding process. Also, traditional methods such as Support Vector Machines (SVMs) and Hidden Markov Models (HMMs) may not match the flexibility and adaptability of neural networks in capturing complex hierarchical representations, limiting their performance in specific contexts. However, they remain robust at handling high-dimensional feature spaces, allowing them to capture complex data patterns better to classify Dysarthric speech.

Gaps and Future Directions

Applying the ML and DL techniques for implementing speech disorders classification, recognition, and prediction models has demonstrated notable improvements in performance accuracy. However, some gaps still need to be addressed as potential avenues for future exploration.

Annotated Datasets

One significant gap lies in the limited availability of large, balanced, and high-quality annotated datasets for training and evaluating machine learning models for speech disorders.^{7,11,12,102,105,117,120,123,138} The scarcity of such datasets limits the generalizability and validity of the results, leading to increased ambiguity and reduced statistical control. Moreover, it hinders the ability to draw strong inferences and develop and evaluate robust algorithms. To develop a robust model, expanding the current state-of-The-art annotation values could be done by adding more human expert annotators, creating a model that would learn an outline according to a selection system, and reproducing several experts’ choices. The less the divergence rate among annotators is, the more efficient the produced device is. Employing post-processing techniques would also enhance the model’s robustness.¹¹²

Diversity

Another limitation is the lack of diversity in the accessible datasets, which hinders the development of more comprehensive and inclusive models. Most available datasets focused on specific languages and communities such as English, French, Spanish, Italian, German, Persian, Korean, Croatian, Indian languages, etc. However, only two datasets provide some level of diversity: Aphasia Bank 25 a. Aphasia Bank comprises English, Croatian, French, Italian, Mandarin, Romanian, and Spanish data. TORGO dataset⁷⁸ covers English and Italian languages. Using one speech database may not represent the diversity of speech disorders and languages and may limit the generalizability and comparison of the results with other types of speech disorders languages.^{117,120,134,138} Moreover, there is a lack of generalization of the machine learning models to other datasets or populations;¹¹ moreover, a lack of comparison between the proposed and existing models for performance, intelligibility, and speech capability loss assessment or accuracy rate prediction.^{11,85,105,135,142} More practice-driven research is required to create standard and large datasets as a feasible approach for researchers to compare different methodologies and techniques, leading to improved results.

Model

In most proposed models, the disordered speech attribute features are based on a binary representation of phonological and phonetic characteristics, which may not capture the fine-grained variations in articulation quality. Moreover, these features are sensitive to noise and recording quality, which may affect the accuracy of anomaly detection and localization and limit their applicability in real-world scenarios.^80,120,133 Most of the proposed models focused on detecting articulation, and much work is still needed for novel models to predict the correct word.¹⁴⁰ More effective models are required while dealing with the extreme variability of speech due to its complex nature.²⁶ These models need to be able to categorize types and severity of speech disorders such as dysarthric or aphasic speech into multiple categories instead of binary classification.^{80,128,135,139,140} However, there are various metrics for evaluating the proposed models’ performance commonly including accuracy in,^{11,12,15,58,63,72,87,90,98,111,128} F1-score in,^14,111 Confidence RMSE in,^87,131 Sensitivity, and Specificity in.⁷⁰ These metrics may not entirely capture the performance of machine learning models in real-world circumstances. There is a need to develop new comprehensive evaluation metrics that are more standardized, allow for better comparison of different models, and capture the nuances and complexities of speech disorders.

Disordered Speech Features

Many studies have claimed that extracting practical acoustic and phonological features is still challenging. Disordered speech features are complex and need special tools and experts in the domain to check and revise their suitability.^125,135,142 Experts should carefully annotate these datasets by extracting phonemic and allophonic features to ensure accurate and reliable training of machine learning models. The Arabic language possesses distinct phonetic sounds, including pharyngeal, larynx, and uvula sounds, often overlooked in speech disorder research. To address this gap, we intend to employ machine learning models to tackle these challenges head-on. Our future studies will be dedicated to exploring the unique phonological characteristics of Arabic speech disorders. Doing so aims to contribute to a more comprehensive understanding of these disorders and pave the way for effective interventions.

Time and Privacy

Most studied approaches focus on the speech recognition phase to recognize wrong keywords, with no focus on deploying such solutions to the final users. Indeed, user profiles are heterogeneous, implying heterogeneity of used datasets for the training step. Even though getting good accuracy from the trained models is related to having a large and shared dataset, this can affect the assistive application’s running time, which needs to operate in real-time. Another issue that can be accurate when using a shared dataset is the privacy concern related to speech disorder patients. As a new direction, overcoming these issues using Federate Learning (FL) techniques can be a promising solution. FL is a machine learning technique recently proposed by Google,²⁸ which is crucial in the era of ubiquitous computing, where massive IoT devices continuously generate relevant data that cannot be easily shared due to privacy and communication constraints. FL is an effective solution for training machine learning models on the growing amount of data while keeping data locale, allowing multiple clients to jointly train a learning model on their private data without revealing their local data to a centralized server.¹³⁰ This can be useful in building an adaptable, trained model for the end-user and ensuring the privacy of sensitive data.

Context of Using the Speech Assessment System

To the best of our knowledge, to predict the correct words, all studied solutions are based on a trained model that is chiefly dependent on the quality of training datasets. To enhance the output efficiency of a speech assessment system for the speech impaired, exploiting the context of the conversation between the patient and his interlocutor can be a new research direction. The context of a conversation can be, for example, the subject of the conversation, the psychological state of the interlocutors, the place and time of the conversation, etc. Detecting and using the context is a challenging task that needs more investigation.

Output of a Speech Assessment System

To our knowledge, all studied solutions mainly focused on detecting incorrect words to propose the correct ones to the end user. In real situations, speech disorder users can pronounce unclear words and sentences with unclear meanings. Dealing with this situation is a challenging task. A large language model such as ChatGPT can be a good direction for predicting a sentence.

Conclusion and Future Work

This work synthesizes and analyses research papers proposing ML-based solutions for speech disorder patients. For this end, we considered a specific number of databases, journals, conferences, and articles published between 2013 and 2023. However, the review only focuses on ML-based papers proposing assistive solutions for people with speech disorder disabilities.

ML-based assistive systems for people with speech disorders are a promising solution for better enhancing the quality of life of these people through communication networking. This work conducted an SLR on ML-based speech disorder assessment systems, aiming to provide a comprehensive understanding of the main issues related to this problem, feature extraction techniques, and ML algorithms used. Our goal was to find out how far we have come and give advice for future research on speech disorders problems. We hope this work will help researchers understand this vital research topic and continue their research.

In the future, we will contribute to this field by considering different issues and trends to follow. Our future work includes two directions. The first direction is to compare different ML-based assistive solutions for speech disorder patients in depth (experimental comparison) to detect their weaknesses and find solutions to remedy them. This will also help academics and practitioners understand how to handle the problem better. Secondly, we aim to propose a new support solution for users suffering from speech disorders by considering the limitations we have detected in existing approaches, such as the limitation of execution time, privacy preservation, diversity of datasets, and the treatment of the Arabic language. This solution will be based mainly on federated learning techniques. Another direction is to consider the user’s emotional state and the subject of the conversation to enhance any assistive solution for users with speech disorders.