Information technologies promise to make products and services more efficient and sustainable, yet the ICT industry has a substantial ecological footprint that is only expected to grow. This calls for actions beyond improving product efficiency. To this end, we explore whether demand for data can be moderated by encouraging ‘sufficient’ usage. We combine behaviour change theories with a critical design approach and conduct a Research through Design exploration of online consumption speculating about possible directions for reducing data use. Partnering with an internet service provider, we designed and tested with end users a set of three provotypes embodying data-sufficient design directions. We reflect on the results to argue that across the entire system of stakeholders, there is insufficient awareness of the environmental impact of data technologies. Further, we conclude by highlighting that contrary to the belief that interventions aimed at reducing consumption harm user experience, our results show that moderating data intake might actually even improve it.@en

This paper provides an empirical and conceptual account on seeing machine learning models as part of a sociotechnical system to identify relevant vulnerabilities emerging in the context of use. As ML is increasingly adopted in socially sensitive and safety-critical domains, many ML applications end up not delivering on their promises, and contributing to new forms of algorithmic harm. There is still a lack of empirical insights as well as conceptual tools and frameworks to properly understand and design for the impact of ML models in their sociotechnical context. In this paper, we follow a design science research approach to work towards such insights and tools. We center our study in the financial industry, where we first empirically map recently emerging MLOps practices to govern ML applications, and corroborate our insights with recent literature. We then perform an integrative literature research to identify a long list of vulnerabilities that emerge in the sociotechnical context of ML applications, and we theorize these along eight dimensions. We then perform semi-structured interviews in two real-world use cases and across a broad set of relevant actors and organizations, to validate the conceptual dimensions and identify challenges to address sociotechnical vulnerabilities in the design and governance of ML-based systems. The paper proposes a set of guidelines to proactively and integrally address both the dimensions of sociotechnical vulnerability, as well as the challenges identified in the empirical use case research, in the organization of MLOps practices.

@en

This paper introduces systems theory and system safety concepts to ongoing academic debates about the safety of Machine Learning (ML) systems in the public sector. In particular, we analyze the risk factors of ML systems and their respective institutional context, which impact the ability to control such systems. We use interview data to abductively show what risk factors of such systems are present in public professionals' perceptions and what factors are expected based on systems theory but are missing. Based on the hypothesis that ML systems are best addressed with a systems theory lens, we argue that the missing factors deserve greater attention in ongoing efforts to address ML systems safety. These factors include the explication of safety goals and constraints, the inclusion of systemic factors in system design, the development of safety control structures, and the tendency of ML systems to migrate towards higher risk. Our observations support the hypothesis that ML systems can be best regarded through a systems theory lens. Therefore, we conclude that system safety concepts can be useful aids for policymakers who aim to improve ML system safety.

@en

This article explores the rapidly developing field of Critical AI Studies and its relation to issues of class and capitalism through a hybrid approach based on distant reading of a newly collected corpus of 300 full-text scientific articles, the creation of which is itself a first attempt at properly delineating the field. We find that words related to issues of class are predominantly but not exclusively confined to a set of studies that make up their own distinct subfield of Critical AI Studies, in contrast to, e.g., issues of race and gender, which are more broadly present in the corpus.

@en

In this article, the following errors have been missed in the corrections stage and the same has been corrected with the correction article. Data Availability section was inadvertently published and it has been removed from the article. The reference Wolters, A. (2022). Guiding the specification of sociotechnical Machine Learning systems: Addressing vulnerabilities and challenges in Machine Learning practice (Unpublished master’s thesis). Massachusetts Institute of Technology is updated to Wolters, A. (2022). Guiding the specification of sociotechnical Machine Learning systems: Addressing vulnerabilities and challenges in Machine Learning practice (Published master’s thesis). Delft University of Technology. Missing equal contribution text and Acknowledgments were included in the original article.

@en

This chapter proposes an analytical lens to comprehensively address the role of Artificial Intelligence (AI) applications in mediating arbitrary exercise of power in public administration and the citizen harms that result from such conduct. It provides a timely and urgent account to fill gaps in conventional Rule of Law thought. AI systems are socio-technical by nature and, therefore, differ from the text-driven social constructs that the legal professions dealing with Rule of Law issues concentrate on. Put to work in public administration contexts with consequential decision-making, technical artefacts can contribute to a variety of hazardous situations that provide opportunities for arbitrary conduct. A comprehensive lens to understand and address the role of technology in Rule of Law violations has largely been missing in literature. We propose to combine a socio-legal perspective on the Rule of Law with central insights from system safety—a safety engineering tradition with a strong scientific as well as real-world practice—that considers safety from a technological, systemic, and institutional perspective. The combination results in a lexicon and analytical approach that enables public organisations to identify possibilities for arbitrary conduct in public AI systems. Following on the analysis, interventions can be designed to prevent, mitigate, or correct system hazards and, thereby, protect citizens against arbitrary exercise of power.@en

This chapter proposes an analytical lens to comprehensively address the role of Artificial Intelligence (AI) applications in mediating arbitrary exercise of power in public administration and the citizen harms that result from such conduct. It provides a timely and urgent account to fill gaps in conventional Rule of Law thought. AI systems are socio-technical by nature and, therefore, differ from the text-driven social constructs that the legal professions dealing with Rule of Law issues concentrate on. Put to work in public administration contexts with consequential decision-making, technical artefacts can contribute to a variety of hazardous situations that provide opportunities for arbitrary conduct. A comprehensive lens to understand and address the role of technology in Rule of Law violations has largely been missing in literature. We propose to combine a socio-legal perspective on the Rule of Law with central insights from system safety – a safety engineering tradition with a strong scientific as well as real-world practice – that considers safety from a technological, systemic, and institutional perspective. The combination results in a lexicon and analytical approach that enables public organisations to identify possibilities for arbitrary conduct in public AI systems. Following on the analysis, interventions can be designed to prevent, mitigate, or correct system hazards and, thereby, protect citizens against arbitrary exercise of power. @en

Algorithmic and data-driven systems are increasingly used in the public sector to improve the efficiency of existing services or to provide new services through the newfound capacity to process vast volumes of data. Unfortunately, certain instances also have negative consequences for citizens, in the form of discriminatory outcomes, arbitrary decisions, lack of recourse, and more. These have serious impacts on citizens ranging from material to psychological harms. These harms partly emerge from choices and interactions in the design process. Existing critical and reflective frameworks for technology design do not address several aspects that are important to the design of systems in the public sector, namely protection of citizens in the face of potential algorithmic harms, the design of institutions to ensure system safety, and an understanding of how power relations affect the design, development, and deployment of these systems. The goal of this workshop is to develop these three perspectives and take the next step towards reflective design processes within public organisations. The workshop will be divided into two parts. In the first half we will elaborate the conceptual foundations of these perspectives in a series of short talks. Workshop participants will learn new ways of protecting against algorithmic harms in sociotechnical systems through understanding what institutions can support system safety, and how power relations influence the design process. In the second half, participants will get a chance to apply these lenses by analysing a real world case, and reflect on the challenges in applying conceptual frameworks to practice.@en

As AI technology becomes more powerful, the impact on governance and the governance of AI becomes more important. The purpose of this panel is to bring together experts on Trustworthy AI and E-Governance to discuss the lessons learned so far and the challenges ahead. The goal is to provide a comprehensive review of challenges and potential solutions related to the future acceptance and beneficial application of AI-based solutions - including using AI for public benefit and applying AI for the common good.@en

Values, such as freedom and safety, are the core motivations that guide us humans. A prerequisite for creating value-aligned multiagent systems that involve humans and artificial agents is value inference, the process of identifying values and reasoning about human value preferences. We introduce a framework that connects the value inference steps, and motivate why a hybrid intelligence approach is instrumental for its success. We also highlight the multidisciplinary research challenges that hybrid value inference entails.

@en

As artificial agents become increasingly embedded in our society, we must ensure that their behavior aligns with human values. Value alignment entails value inference, the process of identifying values and reasoning about how humans prioritize values. We introduce a holistic framework that connects the technical (AI) components necessary for value inference. Subsequently, we discuss how hybrid intelligence'the synergy of human and artificial intelligence'is instrumental to the success of value inference. Finally, we illustrate how value inference both poses significant challenges and provides novel opportunities for multiagent systems research.@en

Algorithmic systems used in public administration can create or reinforce digital cages. A digital cage refers to algorithmic systems or information architectures that create their own reality through formalization, frequently resulting in incorrect automated decisions with severe impact on citizens. Although much research has identified how algorithmic artefacts can contribute to digital cages and their unintended consequences, the emergence of digital cages from human actions and institutions is poorly understood. Embracing a broader lens on how technology, human activity, and institutions shape each other, this paper explores what design practices in public organizations can result in the emergence of digital cages. Using Orlikowski’s structurational model of technology, we found four design practices in observations and interviews conducted at a consortium of public organizations. This study shows that design processes of public algorithmic systems (1) are often narrowly focused on technical artefacts, (2) disregard the normative basis for these systems, (3) depend on involved actors’ awareness of socio-technics in public algorithmic systems, (4) and are approached as linear rather than iterative. These four practices indicate that institutions and human actions in design processes can contribute to the emergence of digital cages, but also that institutional – opposed to technical – possibilities to address their unintended consequences are often ignored. Further research is needed to examine how design processes in public organizations can evolve into socio-technical processes, can become more democratic, and how power asymmetries in the design process can be mitigated.@en

Various AI systems have taken a unique space in our daily lives, helping us in decision-making in critical as well as non-critical scenarios. Although these systems are widely adopted across different sectors, they have not been used to their full potential in critical domains such as the healthcare sector enabled by the Internet of Things (IoT). One of the important hindering factors for adoption is the implication for accountability of decisions and outcomes affected by an AI system, where the term accountability is understood as a means to ensure the performance of a system. However, this term is often interpreted differently in various sectors. Since the EU GDPR regulations and the US congress have emphasised the importance of enabling accountability in AI systems, there is a strong demand to understand and conceptualise this term. It is crucial to address various aspects integrated with accountability and understand how it affects the adoption of AI systems. In this paper, we conceptualise these factors affecting accountability and how it contributes to a trustworthy healthcare AI system. By focusing on healthcare IoT systems, our conceptual mapping will help the readers understand what system aspects those factors are contributing to and how they affect the system trustworthiness. Besides illustrating accountability in detail, we also share our vision towards causal interpretability as a means to enhance accountability for healthcare AI systems. The insights of this paper shall contribute to the knowledge of academic research on accountability, and benefit AI developers and practitioners in the healthcare sector. @en

This chapter formulates seven lessons for preventing harm in artificial intelligence (AI) systems based on insights from the field of system safety for software-based automation in safety-critical domains. New applications of AI across societal domains and public organizations and infrastructures come with new hazards, which lead to new forms of harm, both grave and pernicious. The chapter addresses the lack of consensus for diagnosing and eliminating new AI system hazards. For decades, the field of system safety has dealt with accidents and harm in safety-critical systems governed by varying degrees of software-based automation and decision-making. This field embraces the core assumption of systems and control that AI systems cannot be safeguarded by technical design choices on the model or algorithm alone, instead requiring an end-to-end hazard analysis and design frame that includes the context of use, impacted stakeholders, and the formal and informal institutional environment in which the system operates. Safety and other values are then inherently socio-technical and emergent system properties that require design and control measures to instantiate these across the technical, social, and institutional components of a system. This chapter honors system safety pioneer Nancy Leveson, by situating her core lessons for today’s AI system safety challenges. For every lesson, concrete tools are offered for rethinking and reorganizing the safety management of AI systems, both in design and governance. This history tells us that effective AI safety management requires transdisciplinary approaches and a shared language that allows involvement of all levels of society.@en

As AI systems are integrated into high stakes social domains, researchers now examine how to design and operate them in a safe and ethical manner. However, the criteria for identifying and diagnosing safety risks in complex social contexts remain unclear and contested. In this paper, we examine the vagueness in debates about the safety and ethical behavior of AI systems. We show how this vagueness cannot be resolved through mathematical formalism alone, instead requiring deliberation about the politics of development as well as the context of deployment. Drawing from a new sociotechnical lexicon, we redefine vagueness in terms of distinct design challenges at key stages in AI system development. The resulting framework of Hard Choices in Artificial Intelligence (HCAI) empowers developers by 1) identifying points of overlap between design decisions and major sociotechnical challenges; 2) motivating the creation of stakeholder feedback channels so that safety issues can be exhaustively addressed. As such, HCAI contributes to a timely debate about the status of AI development in democratic societies, arguing that deliberation should be the goal of AI Safety, not just the procedure by which it is ensured.

@en

Real-time data-driven optimization and control problems over networks, such as in traffic or energy systems, may require sensitive information of participating agents to calculate solutions and decision variables. Adversaries with access to coordination signals may potentially decode information on individual agents and put privacy at risk. We use the Inexact Alternating Minimization Algorithm to instantiate local differential privacy for distributed optimization, addressing situations in which individual agents need to protect their individual data, in the form of optimization parameters, from all other agents and any central authority. This mechanism allows agents to customize their own privacy level based on local needs and parameter sensitivities. The resulting algorithm works across a large family of convex distributed optimization problems. We implement the method on a distributed optimal power flow problem that aims to prevent overload on critical branches in a radial network.@en