Adaptive Synthetic Generation of Indefinite Rankings

Abstract

Reducing the similarity of two ranked lists to a single value proves to be useful in various fields of research and industry, such as Information Retrieval and Recommender Systems, leading to the introduction of several similarity measures. One such measure is Rank-Biased Overlap and its variants, possessing qualities such as the ability to handle incomplete rankings, non-conjoint ranking domains and the graceful evaluation of ranking pairs with tied items.
Comparing the performance of similarity measures, or comparing variants of a single measure, requires the presence of ranking data. In certain cases, generating synthetic ranking data may be a more viable option than using real data. However, a review of existing literature reveals a lack of parametrisable synthetic ranking algorithms. This paper introduces a novel method to generate a pair of rankings where one can tailor the conjointness of ranking domains, influence the ranking overlap as a function of depth and tune the presence of tie groups in a probabilistic manner. The paper demonstrates the output of the algorithm when varying the input parameters, verifying the methods performance empirically and statistically.