Dutch State-of-the-art Automatic Speech Recognition (ASR) systems do not perform equally well for different speaker groups. Existing metrics to quantify this bias rely on demographic metadata, which is often unavailable. Recent advances in the field use machine learning to find groups of similar speakers instead. However, its black-box nature obscures the interpretability of resulting groups. This paper proposes an interpretable approach to bias discovery by clustering speakers based on acoustic and prosodic features. Different feature subsets were compared in their ability to find performance disparities in five ASR systems for two separate speaking styles. Results show that these feature sets can uncover bias approaching known disparities between demographic groups. While the effectiveness per feature set differed between the speaking styles, the most successful ones found significant disparities between clusters with diverse demographic compositions.

Automatic Speech Recognition (ASR) systems are becoming increasingly popular in this day and age. Unfortunately, due to inherent biases within these systems, performance disparities exist among specific demographic groups. Bias metrics can be used to measure this bias. Within ASR they represent a niche area that has not yet been thoroughly explored. The few bias metrics that exist in literature mainly centre around the performance differences between speaker groups. This paper proposes two new bias metrics that focus not only on performance differences, but also take the base performance into account: Weighted Performance Bias (WPB) and Intergroup Weighted Performance Bias (IWPB). Although the lack of ground truth makes the results less easily interpretable, the results show similar trends within the new metrics as those defined in literature: bias is greatest among non-native Dutch speech.

Automatic speech recognition (ASR) is increasingly used in daily applications, such as voice-activated virtual assistants like Siri and Alexa, real-time transcription for meetings and lectures, and voice commands for smart home devices. However, studies show that even state-of-the-art (SotA) ASR systems do not recognize the speech of everyone equally well.

To the best of my knowledge, this paper, for the first time, evaluates the performance of Meta's SotA ASR system, Massively Multilingual Speech (MMS), on Dutch native and non-native speech. Using the Jasmin Corpus dataset, which includes a diverse set of both native and non-native Dutch speakers, this study uses metrics such as word error rate (WER), character error rate (CER), and word information lost (WIL) to assess performance. Additionally, the same methodology is applied to the same data using OpenAI's ASR system, Whisper, to provide a comparative analysis.

The paper analyzes WER, CER, and WIL error metrics, processing time, and investigates the best-suited beam size for Whisper. It also lists out the types of errors made in terms of deletions, insertions, and substitutions made by each model across different age groups of Dutch speakers.

Automatic Speech Recognition (ASR) systems have become increasingly important for society, yet their performance varies significantly across different diverse speaker groups. With a significant non-native population in the Netherlands, it is crucial that ASR systems accurately recognize diverse speech. Commercial state-of-the-art ASR systems are yet under-explored in their performance on Dutch diverse speech. This study evaluates the performance of two recently developed and affordable ASR systems, Google Chirp and OpenAI's Whisper, on speech from native and non-native Dutch teenagers. This research evaluates the recognition accuracy of these ASR systems and identifies common transcription errors. The results show slightly worse performance compared to previous research on non-native speech, and Whisper performing generally better than Google Chirp on the speaker groups.

Automatic Speech Recognition is a field that has seen a strong increase in developments in recent years. In order to ensure objectivity and reliability in these systems, it is crucial they remain unbiased and treat speakers equally. This paper explores the bias of two state-of-the-art ASR systems in the domain of Dutch and Flemish speech, specifically towards regional dialects. Specifically, it explores Microsoft's Azure AI Speech Services ASR system and Google Chirp. It analyses the performance of these two systems on the JASMIN-CGN language corpus. The results show that speech from West-Dutch regions is recognized correctly significantly more often than other Dutch regions and speech from Brabant is recognized correctly significantly more often than other Flemish regions.

Automatic Speech Recognition (ASR) technology is becoming more and more useful in everyday life, therefor also requiring higher accuracy across all different user demographics. This study compares the performance of Google's and Microsoft's ASR systems on native Dutch child and teenager speech using the JASMIN-CGN dataset as ASR for children presents unique challenges due to their shorter vocal tracts and irregular speech patterns. This research evaluates each system's performance based on Word Error Rate (WER) and Character Error Rate (CER), highlighting the differences between gender, age, and dialect regions. The results indicate that while Microsoft's ASR consistently outperforms Google's in terms of WER, Google demonstrates slightly higher precision in terms of CER. Therefor Microsoft is considered the better overall performing system but depending on one's needs, such as precision, Google would be the more favorable one.

Automatic Speech Recognition (ASR) systems have become an integral part of daily lives. Despite their widespread use, these systems can exhibit biases that express themselves in the differences in their accuracy and performance across different demographic groups. Methods quantifying these biases have been developed. This paper investigates the relationship between bias and the acoustic characteristics of speakers. By examining various acoustic embeddings, derived from models like wav2vec 2.0 and XLSR, we aim to identify which embeddings correlate most strongly with bias. The findings offer insights into improving the fairness of ASRs by exploring how acoustic features influence bias in ASR systems. Future research directions include exploring isolated speech properties and extending the study to diverse linguistic contexts to deepen understanding in this area.

Automatic Speech Recognition (ASR) systems are found in many places and are used by many people. Some groups of people, superficially older Dutch adults, are recognized less well by these systems. Given the aging population of the Netherlands, it would be beneficial to have ASR systems be more inclusive to allow for more independence of the older adults. By conducting tests on the ASR systems of Google and Microsoft, making use of the JASMIN dataset, I compared the two using word-error-rate (WER), word-information-lost (WIL) and character-error-rate (CER). Results show Microsoft outperforming Google with an average word error rate of 19.6\% compared to 27.35\%. However, Google is less biased on the topics of gender and age. Microsoft was slightly less biased in regards to region, but only by a small margin. Overall, the most notable findings from both systems are a small bias toward female speakers, and a strong bias against speakers from the southern regions of the Netherlands. These findings highlight the need for more inclusive ASR systems, enhancing the independence of older adults.

This paper presents a novel approach to measuring bias in Automatic Speech Recognition (ASR) systems by proposing a metric that does not use the conventional approach of a reference group. Current methods typically measure bias through comparison with a ’norm’ or minimum error group, potentially introducing additional biases. To address this issue, this study introduces a new metric: a combination of the Groupto-Average Log Ratio and the Sum of Group Error Differences. This metric aims to provide a fair comparison by measuring performance relative to the average of the groups rather than a single reference group. Results indicate that the new metric reveals different aspects of bias not captured by traditional methods. This study contributes to the ongoing research on fairness in speech technology by challenging the existing bias metrics and proposing alternatives that might offer more equitable evaluations. Future research should explore further refinements of these metrics, and apply them across more varied datasets and environments. Ultimately, this research moves towards making ASR technologies more inclusive, ensuring that they serve all user groups equitably.

Dysarthric speech, characterized by articulation problems and a slower speech rate, shows lower automatic speech recognition (ASR) performance compared to normal speech. To improve performance, researchers often try to enhance dysarthric speech to be more like normal speech before passing it through an ASR trained on normal speech. In this project, we compare different signal processing and voice conversion techniques for dysarthric-to-normal speech enhancement. The resulting enhanced speech is objectively evaluated using an ASR system trained on normal speech. Also, the naturalness and intelligibility of the enhanced dysarthric speech are evaluated through listening experiments. Finally, the correlation between subjective and objective evaluations was analyzed. We found that among the techniques investigated, time-stretching demonstrated superior performance in objective evaluation experiments, surpassing state-of-the-art voice conversion methods. Across all methods, improvements in naturalness and intelligibility were positively correlated with improvements in automatic speech recognition (ASR) performance. However, this correlation was significant for some methods but not for others.

Brain-Computer Interfaces (BCIs) open avenues for communication among individuals unable to use voice or gestures. Silent speech interfaces are one such approach for BCIs that could offer a trans- formative means of connecting with the external world. Performance on imagined speech decoding however is rather low due to, amongst others, data scarcity and the lack of a clear starting point of the imagined speech in the brain signal. We investigate whether using electroencephalography (EEG) signals from articulated speech can be used to improve imagined speech decoding in two ways: we investigate whether articulated speech EEG signals can be used to predict the end point of the imagined speech and use the articulated speech EEG as extra training data for speaker-independent imagined vowel classification. Our results show that using EEG data from articulated speech did not improve classification of vowels in imagined speech, probably due to high variability in EEG signals amongst speakers.

Abstract— Objective: The objective of this exploratory study is to investigate how AI speech and text technologies, specifically Whisper and ChatGPT-4, can help reduce the administrative burden in occupational health consultations, with a focus on accuracy, efficiency, and user satisfaction.
Methods: A quantitative research approach was employed, utilizing a controlled trial design. Fourteen occupational health doctors participated in simulations using Whisper for transcription and customized ChatGPT-4 modules for generating medical summaries (Medical Summarizer), letters (Letter Generator), and documents (Document Generator), and providing medical protocol-based replies on participants’ questions (Medical Protocol Assistant). The accuracy, efficiency, and user satisfaction of these technologies were compared against traditional administrative methods, with descriptive statistics and paired samples t-tests conducted to assess performance differences.
Findings: The study revealed that Whisper transcriptions had a word error rate (WER) of 13.1%, with an average transcription time of 18.4 minutes, which was 6.6 minutes faster
than human transcription. ChatGPT-4's Medical Summarizer generated summaries 29 times faster than human participants, with an average generation time of 0.5 minutes but had a 38.6%
error rate in element generation. The Letter Generator and Document Generator exhibited error rates of 90.4% and 17.5%, respectively, although both were significantly more efficient
than manual processes, with average generation times of 0.4 and 3.9 minutes, respectively. The Medical Protocol Assistant provided protocol-based replies with an 86.7% accuracy, achieving the highest user satisfaction score (4.4 out of 5) among all modules.
Conclusion: AI speech and text technologies show potential in reducing administrative tasks in occupational health settings, particularly in terms of efficiency. However, the moderate accuracy and varying satisfaction rates indicate that further refinement is necessary to enhance their applicability in clinical practice. Future research should focus on improving accuracy, evaluation of the technologies in actual patient-physician consultations and developing robust privacy safeguards.

Whispering, characterized by its soft, breathy, and hushed qualities, serves as a distinct form of speech commonly employed for private communication and can also occur in cases of pathological speech. The acoustic characteristics of whispered speech differ substantially from normally phonated speech and the scarcity of adequate training data leads to low automatic speech recognition (ASR) performance. This project aims to build an ASR system that can recognize both normal and whispered speech and discover which acoustic characteristics of whispered speech have an impact on whispered speech recognition.
In my study, I use signal processing techniques that transform the spectral characteristics of normal speech to those of pseudo-whispered speech, called pseudo-whispered-based data augmentation. I enhance an End-to-End ASR system by incorporating pseudo-whispered speech and state-of-the-art (SOTA) data augmentation methods, speed perturbation and SpecAugment, yielding an 18.2\% relative reduction in word error rate compared to the strongest baseline.
Results for the accented speaker groups in the wTIMIT database show the best results for US English. Further investigation uncovers that the lack of pitch in whispered speech has the largest impact on the performance of whispered speech ASR.

Patients with neuromuscular diseases that are unable to speak, but whose cognitive ability has been maintained, can be benefited from Brain Computer Interfaces (BCIs). The decoding of inner (covert) speech from EEGs consists of one of the state of the art methods that aim to tackle this issue. High variability between subjects, as well as low signal to noise ratio (SNR) undermine the methods used, and introduce the need for computer assisted solutions. Thus, machine learning models as well as large amounts of recorded data are required to design effective algorithms and produce substantial results. In this study, covert vowel classification was performed in a systematic way, by making use of two openly shared databases from literature; the Coretto database, that contains EEG recordings of native Spanish speakers, and the DAIS dataset, which includes EEG recordings of native Dutch speakers. Six classifiers were initially selected to perform 5-class classification: a Random Forest (RF), a k Nearest Neighbours (kNN), a Gaussian Naive Bayers (GNB), a Deep Convolutional Neural Network (DCNN), a Shallow Convolutional Neural Network (SCNN) and a Long Short Term Memory Recurrent Neural Network (LSTM). The DCNN outperformed the other methods, with average intra-subject accuracies of 35% for Coretto and 39% for DAIS (chance level 20%). Afterwards, an Overt versus Covert trials experiment was implemented, to test the limits of overt speech decoding from EEGs. The overt result was slightly higher than covert, with an intra-subject average value of 37.8% for Coretto and 40.5% for DAIS (chance level 20%). Finally, binary classification was performed to identify those pairs of vowels that can be classified more efficiently. Vowels /a/ and /u/ seemed to perform better in average in both datasets (average of 64.8% for Coretto and 64.4% for DAIS with a chance accuracy of 50%). Future work should focus on identifying the useful parts of the EEG recordings, increasing the SNR and the resolution of the electrodes, and defining the most appropriate dictionaries of words/vowels for a BCI. Also, more studies should follow systematic ways of comparisons between datasets, to obtain less ambiguous insights and lead this field to improvements.

End-to-end Automatic Speech Recognition (ASR) systems improved drastically in recent years and they work extremely well on many large datasets. However, research shows that these models failed to capture the variability in speech production and have biases against the variant caused by the regional accented speech. Moreover, ASR research on regional accents is primarily done in languages used by a large population, like English and Arabic, and the effect of regional accented speech on E2E ASR systems in non-popular languages is still unknown. It is important to know the effect of regional accented speech on E2E ASR systems as it helps researchers to build an inclusive E2E ASR system. In this project, I aim to mitigate the biases against regional accented speech. I select standard speech and regional accented speech from CommonVoice's French and German datasets. I combine the state-of-the-art Conformer Recurrent Neural Network Transducer model with Multi-Domain Adversarial Training (MDAT) to boost the performance of regional accented speech while not hurting the performance of the standard speech. Moreover, since the regional accented speech is typically low-resourced, I study the amount of data required for effective MDAT, as well as the effect of different domain classifiers on the performance of Multi-Domain Adversarial Training. Experimental results show that MDAT can mitigate the biases against regional accented speech in both French and German. The best model in French reduces the bias by around 12% and the best model in German reduces the bias by around 7%. Additionally, MDAT is an effective method for bias mitigation as it can achieve similar performance as the MDAT model trained with the full dataset using only a small amount (e.g. 30 minutes) of untranscribed regional accented speech. Finally, different domain classifier architectures were found to have similar effects on the results of MDAT, thus there is no significant differences among the domain classifier in this project.

The field of speech-based Personality Computing classifies personality traits using speech data. There are two labelling methods for this: Automatic Personality Recognition (APR), using self-assessed personality scores, and Automatic Personality Perception (APP), using externally rated personality scores. Another aspect is whether the data is recorded in natural circumstances or in a controlled environment, as this influences how personality is shown. There is a lack of research into these speech styles, especially when combined with the labelling methods. Related fields have been found to be more developed in two ways. First, research from the perspective of speech styles has already been conducted and proven useful. Second, when state-of-the-art techniques are released, such as pretrained models targetting speech, these fields are often included in benchmark tests. As no personality datasets are included, this creates a knowledge gab on using these techniques for personality classification.

The influence of the labelling methods and speech styles is investigated using three datasets that target APR with controlled and natural speech, and APP with natural speech.
Three types of models are used to see what personality traits can successfully be classified. The two APR datasets have not been used for personality classification before. Additionally, the models are trained using both hand-crafted features and embeddings extracted from a state-of-the-art pretrained model. The experiment on the APP and natural speech dataset indicates that the performance for 3 out of 5 traits can be improved using more effective features. The APR and controlled speech dataset was able to classify 4 out of 5 traits above chance. The APR and natural speech dataset could not well be classified. Overall, the APR datasets performed worse than the APP dataset. There were no clear patterns found between the speech styles. Furthermore, the embeddings showed better overall performance than the hand-crafted features. Future work could standardize a dataset for both labelling methods and speech styles to make direct comparison between the methods possible.

In this thesis, we analyzed and compared speech representations extracted from different frozen self-supervised learning (SSL) speech pre-trained models on their ability to capture articulatory feature (AF) information and their subsequent prediction of phone recognition performance in within-language and cross-language scenarios. Specifically, we compared speech representations of three SSL speech pre-trained models, including CPC, wav2vec 2.0, and HuBert. Firstly, frame-level AF probing tasks were implemented to analyze AF information captured by different speech representations. Subsequently, phone-level ASR systems were implemented to analyze the phoneme recognition performance of these speech representations. Results showed that the performance of the frame-level AF probing task and the accuracy of the phoneme recognition task were correlated. Compared to the conventional speech representation MFCC, all SSL pre-trained speech representations captured more AF information and achieved better phoneme recognition performance in within-language and cross-language scenarios, with HuBert performing best. Meanwhile, the frame-level AF probing task is a good predictor of phoneme recognition performance, showing the importance of capturing AF information in speech representations. Compared with MFCC, in the within-language scenario, the performance of these SSL speech pre-trained models on AF probing tasks achieved a maximum relative increase of 34.4%, and it resulted in the lowest PER of 10.2%. In the cross-language scenario, the maximum relative increase of 26.7% resulted in the lowest PER of 23.0%.

Automatic speech recognition (ASR) does not perform equally well on every speaker. There is bias against many attributes, including accent. To train Dutch ASR, there exists CGN(Corpus Gesproken Nederlands) and as an extension, the JASMIN corpus with annotated accented data. This paper focuses on improving ASR performance for NRAD (Northern regional accented Dutch) speech, training on speakers from the region of Overijssel. To achieve this improvement, the corpus data is augmented using Vocal Tract Length Perturbation (VTLP), which entails randomly warping the frequency of each recording using a factor in the range [0.9, 1.1]. The baseline and augmented ASR systems are trained using trigram GMM-HMM (Gaussian mixture model hidden Markov models) through the Kaldi toolkit on the DelftBlue supercomputer. This leads to improvements on word error rates (WER) for all speaker groups and styles, with an overall relative improvement of 14,64% and the biggest improvement observed for male speakers - from 25.15% WER to 19,68% WER. The impact of this augmentation on other accents and non-accented speech is not explored. This experiment can serve as a stepping stone for developing overall more robust and less biased Dutch ASR.

There are many experiments conducted with Automatic Speech Recognition (ASR) systems, but many either focus on specific speaker categories or on a language in general. Therefore, bias could occur in such ASR systems towards different genders, age groups, or dialects. But, to analyze and reduce bias, the models require significant amounts of data to be trained on, and some corpora lack that. This is where augmentation techniques can be used to generate more unique data without any further collection of it. This paper explores the use of SpecAugment's frequency masking on such a corpus, JASMIN-CGN, for the Transitional regional accent of Dutch, with a hybrid GMM-HMM architecture, in order to reduce the bias for gender or age, for this specific dialect. The experiments show that SpecAugment does not manage to lower the WER (20.8% overall compared to the baseline model, which achieves 19.5% performance), on the contrary, it even increases the bias for age. The results are mainly attributed to the combination of low amounts of data + the hybrid architecture used, which proves SpecAugment to be a useful augmentation policy only for end-to-end models.

Building Automatic Speech Recognizers (ASRs) has been a challenge in languages with insufficiently sized corpora or data sets. A further large issue in language corpora is biases against regionally accented speech and other speaker attributes. There are some techniques to improve ASR performance and reduce biases in these corpora, known as data augmentations. One audio data augmentation, pitch shifting, has had successes in other experiments for increasing ASR performance. Pitch shifting it is tested in this paper on the JASMIN-CGN speech data set from the Southern regions of the Netherlands. Using a hybrid GMM-HMM ASR, two baselines are developed, one using all speech data from the region, the other only using native speech. For the former ASR, pitch shifting is found to not improve Word Error Rate (WER) performance or reduce bias, but the latter succeeds in improving WER performance and reduced bias for certain speaker groups when augmented.