This paper introduces systems theory and system safety concepts to ongoing academic debates about the safety of Machine Learning (ML) systems in the public sector. In particular, we analyze the risk factors of ML systems and their respective institutional context, which impact the ability to control such systems. We use interview data to abductively show what risk factors of such systems are present in public professionals' perceptions and what factors are expected based on systems theory but are missing. Based on the hypothesis that ML systems are best addressed with a systems theory lens, we argue that the missing factors deserve greater attention in ongoing efforts to address ML systems safety. These factors include the explication of safety goals and constraints, the inclusion of systemic factors in system design, the development of safety control structures, and the tendency of ML systems to migrate towards higher risk. Our observations support the hypothesis that ML systems can be best regarded through a systems theory lens. Therefore, we conclude that system safety concepts can be useful aids for policymakers who aim to improve ML system safety.

@en

As the application of machine learning (ML) algorithms becomes more widespread, governmental organisations try to benefit from this technology. While ML has the potential to support public services, its application also introduces challenges. Several scholars have described the possible opportunities and challenges of ML applications in the public sector conceptually. However, it is not yet investigated if and how these concepts materialise and are perceived by end-users in the public sector when ML is applied. Therefore, it is neither clear whether these concepts are valid, nor what regulation could be introduced to address them effectively. This empirical study's objective is to shed light on how challenges and opportunities of governmental use of ML algorithms are perceived by Dutch professionals in the public sector. We attain our objective by conducting interviews with twelve professionals from Dutch executive and supervisory organisations in the public sector that respectively use ML and supervise the use of ML. Results show that ML is used primarily for improvements in the accuracy and speed of public task execution. Furthermore, interviewed professionals experience several barriers for ML implementation as well as risks following from the use of ML. The implications of these findings for practice are discussed, as well as opportunities for further research. @en