Publications
CONFERENCE (INTERNATIONAL) Detecting Absurd Conversations from Intelligent Assistant Logs by Exploiting User Feedback Utterances
Chikara Hashimoto and Manabu Sassano
The Web Conference 2018 (WWW 2018)
April 23, 2018
Intelligent assistants, such as Siri, are expected to converse comprehensibly with users. To facilitate improvement of their conversational ability, we have developed a method that detects absurd conversations recorded in intelligent assistant logs by identifying user feedback utterances that indicate users' favorable and unfavorable evaluations of intelligent assistant responses; e.g., "great!" is favorable, whereas "what are you talking about?" is unfavorable. Assuming that absurd/comprehensible conversations tend to be followed by unfavorable/favorable utterances, our method extracts some absurd/comprehensible conversations from the log to train a conversation classifier that sorts all the conversations recorded in the log as either absurd or not. The challenge is that user feedback utterances are often ambiguous; e.g., a user may give an unfavorable utterance (e.g., "don't be silly!") to a comprehensible conversation in which the intelligent assistant was attempting to make a joke. An utterance classifier is thus used to score the feedback utterances in accordance with how unambiguously they indicate absurdity. Experiments showed that our method significantly outperformed methods that lacked a conversation and/or utterance classifier, indicating the effectiveness of the two classifiers. Our method only requires user feedback utterances, which would be independent of domains. Experiments focused on chitchat, web search, and weather domains indicated that our method is likely domain-independent.
Paper :
Detecting Absurd Conversations from Intelligent Assistant Logs by Exploiting User Feedback Utterances
(external link)