Audio-Visual Speech Recognition in MISP2021 Challenge

Dataset Release and Deep Analysis

Journal article (2022)

Authors

Hang Chen University of Science and Technology of China
Jun Du University of Science and Technology of China
Yusheng Dai University of Science and Technology of China
Chin-Hui Lee Georgia Institute of Technology
Sabato Marco Siniscalchi University of Enna Kore, Georgia Institute of Technology
Shinji Watanabe Carnegie Mellon University
Bao Cai Yin iFlytek
Jia Pan iFlytek
Copyright: © 2022 Hang Chen, Jun Du, Yusheng Dai, Chin Hui Lee, Sabato Marco Siniscalchi, Shinji Watanabe, O.E. Scharenborg, Jingdong Chen, Bao Cai Yin, Jia Pan
DOI: https://doi.org/10.21437/Interspeech.2022-10483
More Info
expand_more

Published Date

2022

Language

English

Reuse Rights

Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.

Abstract

In this paper, we present the updated Audio-Visual Speech Recognition (AVSR) corpus of MISP2021 challenge, a large-scale audio-visual Chinese conversational corpus consisting of 141h audio and video data collected by far/middle/near microphones and far/middle cameras in 34 real-home TV rooms. To our best knowledge, our corpus is the first distant multi-microphone conversational Chinese audio-visual corpus and the first large vocabulary continuous Chinese lip-reading dataset in the adverse home-tv scenario. Moreover, we make a deep analysis of the corpus and conduct a comprehensive ablation study of all audio and video data in the audio-only/video-only/audiovisual systems. Error analysis shows video modality supplement acoustic information degraded by noise to reduce deletion errors and provide discriminative information in overlapping speech to reduce substitution errors. Finally, we also design a set of experiments such as frontend, data augmentation and end-to-end models for providing the direction of potential future work. The corpus and the code are released to promote the research not only in speech area but also for the computer vision area and cross-disciplinary research.

Files

Chen22o_interspeech.pdf
(pdf | 7.73 Mb)
- Embargo expired in 01-07-2023
Unknown license