Enriching Machine Learning Model Metadata

Abstract

As the sharing of machine learning (ML) models has increased in popularity, more so-called model zoos are created. These repositories facilitate the sharing of models and their metadata, and other people to find and re-use an existing model. However, the metadata provided for models is insufficient, with little focus on practical aspects of a model such as performance and limitations. This leaves model zoos to provide little functinality beyond the sharing of existing models, and potential users unable to find an optimal model for their needs. In this thesis we focus on the reporting of model performance, and aim to answer two questions: (1) What are the limitations of model performance reporting found in current model zoos, and (2) How can we provide rich, comparable performance metadata for these models. As a result, we have created a framework which can be used to create a benchmarking system for ML models. Our framework enables the automatic evaluation of models from any source, and through its design encourages the collection of rich and comparable performance metadata for those models. To demonstrate the effectiveness of our framework, we have created a benchmarking system and evaluated 1215 models from existing model zoos, along with a new interface to view this new metadata, and have enabled new use-cases such as the performance comparison of ML models.