Dealing with overconfidence and bias in low-cost evaluation of audio music similarity

Abstract

The music information retrieval community has long recognized the need for better evaluation. Frameworks as comprehensive as in the field of text retrieval are missing. An important problem of evaluation for music retrieval is the high cost. This work continues on the proposed low-cost evaluation model. The low-cost model allows for cheap evaluation with a degree of uncertainty. Alongside the evaluation results, confidence is indicated for interpretability. The confidence is however overconfident. This work breaks down the causes and introduces solutions that reduce this overconfidence. The evaluation results of certain metrics are structurally overestimated. The source of this problem is found and an appropriate solution is explained.