Publications
CONFERENCE (INTERNATIONAL) Online Predictive Coding for Dual-Mode Self-Supervised Speech Models
Keita Goto, Takashi Maekaku, Jin Sakuma, Jinchuan Tian (Carnegie Mellon University), Yusuke Shinohara, Shinji Watanabe (Carnegie Mellon University)
The 27th Annual Conference of the International Speech Communication Association (INTERSPEECH 2026)
September 28, 2026
Dual-mode self-supervised speech models are pre-trained to handle streaming and non-streaming conditions simultaneously. However, their attention is computed over different context ranges, which often makes optimization difficult. In previous work, we proposed online registers, additional tokens intended to compensate for missing future context in streaming mode, but the gains remained limited. To address these issues, we introduce two improvements for robust dual-mode pre-training: (1) Online Predictive Coding (OPC), which regularizes the registers through multi-step future prediction, and (2) Dual-mode Layer Normalization, which stabilizes optimization. We fine-tune the proposed dual-mode self-supervised speech models for speech recognition on LibriSpeech and WSJ. Results show that OPC consistently reduces the online–offline performance gap; at 160 ms latency on LibriSpeech, word error rates improve from 3.65% to 3.40% on test-clean and from 10.15% to 9.65% on test-other.
Paper :
Online Predictive Coding for Dual-Mode Self-Supervised Speech Models
(external link)
PDF : Online Predictive Coding for Dual-Mode Self-Supervised Speech Models