Publications
CONFERENCE (INTERNATIONAL) Exploring Speech Recognition, Translation, and Understanding with Discrete Speech Units: A Comparative Study
Xuankai Chang (Carnegie Mellon University), Brian Yan (Carnegie Mellon University), Kwanghee Choi (Carnegie Mellon University), Jee-Weon Jung (Carnegie Mellon University), Yichen Lu (Carnegie Mellon University), Soumi Maiti (Carnegie Mellon University), Roshan Sharma (Carnegie Mellon University), Jiatong Shi (Carnegie Mellon University), Jinchuan Tian (Carnegie Mellon University), Shinji Watanabe (Carnegie Mellon University), Yuya Fujita, Takashi Maekaku, Pengcheng Guo (Northwestern Polytechnical University), Yao-Fei Cheng (University of Washington), Pavel Denisov (University of Stuttgart), Kohei Saijo (Waseda University), Hsiu-Hsuan Wang (National Taiwan University)
2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2024)
March 20, 2024
Speech signals, typically sampled at rates in the tens of thousands per second, contain redundancies, evoking inefficiencies in sequence modeling. High-dimensional speech features such as spectrograms are often used as the input for the subsequent model. However, they can still be redundant. Recent investigations proposed the use of discrete speech units derived from self-supervised learning representations, which significantly compresses the size of speech data. Applying various methods, such as de-duplication and subword modeling, can further compress the speech sequence length. Hence, training time is significantly reduced while retaining notable performance. In this study, we undertake a comprehensive and systematic exploration into the application of discrete units within end-to-end speech processing models. Experiments on 12 automatic speech recognition, 3 speech translation, and 1 spoken language understanding corpora demonstrate that discrete units achieve reasonably good results in almost all the settings. We intend to release our configurations and trained models to foster future research efforts.
Paper :
Exploring Speech Recognition, Translation, and Understanding with Discrete Speech Units: A Comparative Study
(external link)