Publications
WORKSHOP (INTERNATIONAL) Worker Viewpoints: Valuable Feedback for Microtask Designers in Crowdsourcing
Ryota Hayashi (University of Tsukuba), Nobuyuki Shimizu, Atsuyuki Morishima (University of Tsukuba)
The International Workshop on Social Computing (IWSC 2017)
April 04, 2017
One of the problems a requester faces when crowdsourcing a microtask is that, due to the underspecifie or ambiguous task description, workers may misinterpret the microtask at hand. We call a set of such interpretations worker viewpoints. In this paper, we argue that assisting requesters to gather a worker’s interpretation of the microtask can help in providing useful feedback to designers, who may restate the task description if necessary. In our method, we create a corpus of viewpoints annotated with the types of viewpoints that reflec the logical structure embedded in them. Our experimental results suggest that the logic-oriented annotation is effective in choosing useful viewpoints from a possibly huge set of collected viewpoints, in the sense that removing viewpoints of particular types did not affect the quality of revised task instructions. We also show that the logic-oriented annotation can perform comparably with an entropy-based method, without several workers performing the same task in parallel.
Paper :
Worker Viewpoints: Valuable Feedback for Microtask Designers in Crowdsourcing
(external link)