Evaluating Methods for Improving Crowdsourced Annotations of Images Containing Lego Bricks

Abstract

Data collection by means of crowdsourcing can be costly or produce inaccurate results. Methods have been proposed for solving these problems. However, it remains unclear what methods work best in scenarios with multiple similar objects of interest present in the same image, which is important for training computer vision with applications such as automatic quality control in factories. We researched which parameters are important to optimize, which methods are worth considering and what those selected methods score with regard to the parameters cost and quality. This was done through a literature review and substantiated by an experimental crowdsourcing campaign that focused on the annotation of Legos in images. It was found that the parameters to optimize were cost, optimized by reducing the time workers spent on tasks, and quality, optimized by improving the mean intersection over union value of the annotations. We concluded that majority vote, rejecting workers, majority vote adjusted to be resistant to outliers, rejecting workers with the same adjustments and decomposing tasks were the most promising methods. From our experiment we concluded that a clear trade-off exists between cost and quality. The adjusted rejecting workers method, that uses worker credibility, showed to have the highest mean quality. While the method that decomposed the components of the task and distributed them was the cheapest method to use overall and also best when looking at mean quality over cost, it was worse quality wise. These results were similar to the expected performance of the methods. From this we concluded that the best method for crowdsourcing is dependent on the error tolerance of the computer vision model that will be used and the budget available.