Retrieving Collective Distribution of Crowd Annotation by Estimating How Others Would Annotate

less than 1 minute read

Crowdsourcing has been widely used for labeling categories of data. It has tried to aggregate varied and error-prone responses into one accurate category. However, some data like the audio-visual expression of the emotion can be interpreted in diverse ways. For such data, the overall perception of the crowd on the data can convey how diverse interpretations are possible. However, how to collect such distribution of the perception efficiently is under-explored and naive crowdsourcing method can be expensive to collect the distribution. In this paper, we explore how we can crowdsource the perception of the overall crowd efficiently by leveraging ‘crowd within’ or individual workers giving richer information on diverse interpretations of the data. We gradually expand the information given by crowd workers, from one best answer to all possible answers, the different confidence for all selected answers, and the estimation of how other crowd workers would perceive the data.