Publications
CONFERENCE (INTERNATIONAL) LSTM vs. BM25 for Open-domain QA: A Hands-on Comparison of Effectiveness and Efficiency
Sosuke Kato (Waseda University), Riku Togashi , Hideyuki Maeda, Sumio Fujita, Tetsuya Sakai (Waseda University)
The 40th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2017)
August 15, 2017
Recent advances in neural networks, along with the growth of rich and diverse community question answering (cQA) data, have enabled researchers to construct robust open-domain question answering (QA) systems. It is often claimed that such state-of-the-art QA systems far outperform traditional IR baselines such as BM25. However, most such studies rely on relatively small data sets, e.g., those extracted from the old TREC QA tracks. Given massive training data plus a separate corpus of Q&A pairs as the target knowledge source, how well would such a system really perform? How fast would it respond? In this demonstration, we provide the attendees of SIGIR 2017 an opportunity to experience a live comparison of two open-domain QA systems, one based on a long short-term memory (LSTM) architecture with over 11 million Yahoo! Chiebukuro (i.e., Japanese Yahoo! Answers) questions and over 27.4 million answers for training, and the other based on BM25. Both systems use the same Q&A knowledge source for answer retrieval. Our core demonstration system is a pair of Japanese monolingual QA systems, but we leverage machine translation for letting the SIGIR attendees enter English questions and compare the Japanese responses from the two systems after translating them into English.
Paper :
LSTM vs. BM25 for Open-domain QA: A Hands-on Comparison of Effectiveness and Efficiency
(external link)