Web surveys have increasingly been used to collect data from respondents over the years. They offer several advantages compared to other methods of obtaining data. Researchers benefit from a broad demographic representation to make generalized conclusions, and satisfaction surveys allow employees to explain shortcomings or improvements anonymously. Both examples demand comprehensive information, thereby requiring a lengthy survey. However, dropout increases with the length of a survey, which is a big problem on web surveys as it decreases the statistical significance of the results. Proposed solutions, such as reducing the number of questions or rewarding respondents with an incentive, may not always be feasible due to the preciseness of information required or limited financial capabilities.

Achievement primes have been shown to reduce dropout on short surveys targeting extrinsically motivated respondents without additional costs or the need to reduce survey length. As repeated exposure to primes reinforces the stimuli, long surveys may also benefit from achievement primes. In this study, respondents are exposed to a questionnaire of more than 15 minutes on health whilst working behind a computer containing either no prime, passive achievement primes, or active achievement primes. Besides extrinsically motivated respondents, recruited via the crowdworking platform Prolific, intrinsically motivated respondents are also targeted in this study, recruited via snowball sampling.

Through a 2 times 3 factorial design, we discovered no statistical difference in dropout, perceived workload, and user engagement across the three questionnaire variants when evaluating intrinsically (N=88) and extrinsically motivated respondents (N=140) individually. By comparing intrinsically with extrinsically motivated respondents, we discovered extrinsically motivated respondents were more engaged and dropped out less.

Machine Learning models are nowadays infused into all aspects of our lives. Perhaps one of its most common applications regards recommender systems, as they facilitate users' decision-making processes in various scenarios (e.g., e-commerce, social media, news, online learning, etc.). Training performed on large volumes of data is what ultimately drives such a system to provide meaningful recommendations, and yet there has been observed a lack of standardized practices when it comes to data collection and annotation methods for Machine Learning datasets. This research paper systematically identifies and synthesizes such processes by examining existing literature on recommender systems. The review includes 100 most-cited papers from the most impactful venues within the Computing and Information Technology field. Multiple facets of the employed techniques are touched upon, such as reported human annotations and annotator diversity, label quality, and the public availability of training datasets.
Recurrent use of just a few benchmark datasets, poor documentation practices, and reproducibility issues in experiments are some of the most striking findings uncovered by this study. A discussion is centered around the necessity of transitioning from reliance solely on algorithmic performance metrics in favor of prioritizing data quality and fit. Finally, valid concerns are raised when it comes to biases and socio-psychological factors inherent in the datasets, and further exploration of embedding these early in the design of ML models is suggested.

This research examines transparency between ICASSP conference papers and the dataset documentations related to the datasets' annotation practices. Top-cited 5 papers and 51 unique resources in total were considered. All of the selected papers utilized at least one dataset. For every paper, an extensive metadata search has done to reach the initial datasource of those datasets. These searches happened both within the paper contents such as sections and references along with outside the paper contents through the way of extensive web queries. Analysis of the papers published from 2021 and 2022 and their relevant datasets revealed varying levels of transparency. Original dataset creators provide comprehensive information, while papers using modified datasets offer limited details on initial annotations. Emphasizing the need for accountability, this study suggests that papers utilizing datasets should trace back to the initial dataset and provide explicit comments. The findings underscore the importance of ensuring sufficient information in initial datasets and promoting transparency and traceability in dataset annotation practices within the ICASSP community.

Depression diagnosis and treatment remain difficult tasks that could be improved with machine learning models. But those automatic systems should be reliable to apply in clinical psychology settings. Performing predictions in this field is most commonly done using supervised learning models, which rely on well-established annotations. Therefore this paper examines annotation practices in research surrounding depression and provides a perspective on the quality of established methods. Firstly, 80 papers were surveyed in terms of reported annotation practices. Then the results were collected and analyzed. The findings suggest that papers from the Computer Science domain would benefit from the utilization of expert knowledge and better practices of human verification. While papers across all domains missed important information on the annotation process and rarely (20\% of papers) provided input data.

In the machine learning research community, significant importance is given to the optimization of techniques which are employed once a benchmark dataset is given. However, less importance is assigned to the quality of these datasets and to how these datasets are obtained. In this work, we look into annotation practices in the research area of affective computing, analysing datasets of emotion classification tasks from text, video, audio, EEG data and more. We find annotation practices of varying quality and recommend that annotation practices be improved, especially with regard to multiple annotator overlap.

This systematic review investigates the practices and implications of human annotations in machine learning (ML) research. Analyzing a selection of 100 papers from the IEEE Access Journal, the study explores the data collection and reporting methods employed. The findings reveal a prevalent lack of standardization and formalization in the annotation process. Key details such as annotation sources, number of annotators, and formal instructions are frequently neglected, possibly compromising the quality and effectiveness of ML algorithms. Domain-specific implications are discussed, highlighting the need for comprehensive annotation practices in areas like medical diagnostics, language processing, and intelligent vehicle systems. The study contributes to the field by emphasizing the importance of standardized procedures and transparency in ML research. Future research is recommended to develop systematic annotation methodologies and examine the impact of subpar annotation on data quality.

Longer lifespans and an ageing population put tremendous pressure on the care of the elderly. With the technology of robotics breakthroughs, it appears that the use of robotics in elderly care is ready to take off. Interestingly, more and more robots are now being created with a human-like appearance and demeanour, yet this could lead to issues like inappropriate mental models of robots in the user's mind. On the other hand, due to their minimal communication challenges, robot-like robots may function effectively in the human-robot interaction. Up to now, far too little attention has been paid to the increasing trend of the human-like robot. This thesis compared the performance of these two types of robots, which may help to explain why the trend is the way it is. Two research questions are created to accomplish this goal: Which type of robot, robot-like or human-like, produces the most fluent interaction in the elderly care scenario? Does the robot-like or human-like robot bring about the best memory performance in the music listening activity?

This thesis project aims to contribute to the development of the social robot in elderly care. In this thesis, a music assistance robot was developed, it can converse with the user while listening to music and recalling memories associated with the song. The elder's health and well being are benefited by this pastime. To address the question of which type of robot performs better, two types of robots, a human-like robot and a robot-like robot, were created to compare in terms of fluency in human-robot interactions and performance of memory recall during music listening activities. Four key elements: appearance, behaviour, voice and dialogue were used in the construction of two robot identities. Based on the robot we created, a joint music listening experiment was carried out. For evaluation, a questionnaire and video recordings of the interaction were both employed. This was tested with 30 volunteers (The young adults were chosen for the experiment due to COVID restrictions and time limitation), and it was discovered that the human-like robot performed better overall.

Several input types have been developed in different technological landscapes like crowdsourcing and conversational agents. However, sign language remains one of the input types that has not been looked upon. Although numerous amount of people around the world use sign language as their primary language, there have not been many efforts to include them in these technological landscapes. In this thesis, we hope to draw attention to and take a step towards the inclusion of deaf and mute people in microtask crowdsourcing.
We identify some of the existing technical and research gaps in the current architectures for Sign Language Recognition/Translation in a real-time setting. Next, we determine various microtasks which can be adapted to use sign language as input, keeping in mind the challenges it introduces. We, then, investigate the effectiveness of a system that uses sign language as input by building a web application - SignUpCrowd - for microtask crowdsourcing, namely Visual Question Answering and Tweet Sentiment Analysis tasks, and comparing it with already prevalent input types such as text and click. This comparison with different popular input types will help understand how much of a difference there is for sign language as input. In addition, it will also show the preference of input types for the particular microtasks. For this, we developed three web applications with different input types and conducted a between-subject experimental study on Prolific wherein a number of workers (N=240) were asked to perform the above-mentioned tasks using sign language, text, and click input. Our results indicate that, in terms of task completion time and task accuracy, sign language as an input modality in microtask crowdsourcing is not significantly different from other, commonly used, input types. We also noticed that people's input type preference for the given microtasks for sign language was more than text input. Although people with no knowledge of sign language found it difficult, this input modality aims at a different target audience. This shows us that there is scope for sign language as an input type for microtask crowdsourcing among people, and paves the way for more efforts for the introduction of sign language in real-world applications.

The ability to identify and mitigate various risks and harms of using Machine Learning models in industry is an essential task. Specifically because these may produce harmful outcomes for stakeholders, including unfair or discriminatory results. Due to this there has been substantial research into the concepts of fairness and its metrics, bias and its mitigation, and algorithmic harms and their sources. Various toolkits have been created to guide practitioners to reflect on these topics and provide suggestions on algorithmic solutions to mitigate these risks. However, it is not yet known how widely these toolkits are used and how they are perceived in terms of usefulness. In this project, practitioners were interviewed in order to determine to what extend do envisioned practices of practitioners without experience with fairness toolkits differ from those with the experience. The two toolkits considered were the IBM AI Fairness360 and Microsoft FairLearn. The data collected from the interviews suggests that there could be fewer differences in practices of practitioners with experience and without experience with toolkits, than those with training or work roles in ethics and fairness in ML and those without. This suggests that experience the toolkit itself is not indicative of a more thorough approach to identifying and mitigating harms in fair Machine Learning.

The increasing dangers of unfairness in machine learning (ML) are becoming a frequent subject of discussion, both, in academia and popular media. Recent literature focused on introducing and assessing algorithmic solutions to bias in ML. However, there is a disconnect between these solutions and practitioners' needs. By interviewing 30 ML practitioners of diverse backgrounds across 16 countries, and presenting them with a simulated use case, our study aims to investigate common fairness practices among professionals and how these are influenced by their backgrounds. The results reveal a superlative disparity among academia and industry practitioners. We also identify different practices in fairness and data exploration stages, influenced by the educational background as well as the level of experience of practitioners. Our study also finds how demographics have an impact on several aspects, such as willingness to accept and support legal actions taken against ML discrimination. In accordance with our findings, we suggest several actions that can be taken to improve fairness solutions, and we also highlight future directions for fairness research that can cause a positive impact on the way fairness is perceived by practitioners.

Machine learning is still one of the most rapidly growing fields, and is used in a variety of different sectors such as education, healthcare, financial modeling etc(Jordan and Mitchell 2015). However, along with this demand for machine learning algorithms, there comes a need for ensuring that these algorithms are fair and contain little to no bias. Tools like Fairlearn and AI Fairness 360(AIF360) allows developers and data scientists to examine their codebase according to specified fairness metrics and mitigate any fairness related issues. This study aims to determine how practitioners use the separate toolkits and whether their practices are differ by the toolkit they choose to use. To do this, we conducted 29 think-aloud interviews with industry practitioners to understand how they would use Fairlearn and AIF360 in practice. The results show that fairness is a socio-technical challenge. While the toolkit does allow for participants to be structured in their approach, and raise awareness for fairness related harms, at the end of the day the fairness toolkit only provides technical help to find harms that the individual was already aware about. Based on the findings, we then suggest the design for a fairness toolkit that can help practitioners approach fairness in the most ideal manner. This toolkit would include a way to have interdisciplinary collaboration, have a larger focus on explainability, and give clear guidance to its users regarding fairness related harms.

To encourage ethical thinking in Machine Learning (ML) development, fairness researchers have created tools to assess and mitigate unfair outcomes. However, despite their efforts, algorithmic harms go beyond what the toolkits currently allow to measure. Through 30 semi-structured interviews, we investigated whether data scientists are constrained to only thinking about issues that can be tackled with these toolkits when using them in practice. The results of a comparative assessment of approaches with and without a toolkit indicate that although they can be incredibly effective, toolkits shouldn't replace educating on sources of harm and can even have hazardous consequences when improperly used. We discovered that while fairness toolkits increase practitioners' awareness of several specific sources of harm, such as questionable attributes or data sampling techniques, their greater power lies in fostering discussions about ML systems' propensity to treat individuals unfairly. On the contrary, we observed that these toolkits do not significantly help in the data documentation process, and, from observing our study participants, we also infer a risk of them blindly evaluating and optimizing for undesired outcomes as a result of choosing metrics and mitigations on unfounded or incomplete assumptions. This work supports future improvement of toolkits by providing a breakdown of perspectives around various sources of harm and reasoning about the ones that get frequently overlooked.

Research that focuses on examining software bugs is critical when developing tools for preventing and for fixing software issues. Previous work in this area has explored other types of systems, such as bugs of compilers and security issues stemming from open source systems hosted on public repositories. This paper explores the bugs within the Ansible software provisioning and configuration management system. The main question this paper seeks to answer is "What common patterns can be extracted from the bugs found and what are the root causes, symptoms, triggers, system-dependence factors, fixes, and the impact of the most frequent types of bugs in the Ansible configuration management system". This study defines a data pipeline and custom tools to extract and analyze 100 Ansible bugs. Common classifications are determined, and the bugs are manually classified, revealing common patters within the bugs. Insights are drawn from the aggregated data, and recommendations are made for addressing bug-prone areas of execution and connectivity components, and expanding the test suite with input fuzzing and a genetic algorithms test solution, in order to improve the overall code quality of the Ansible code base.

This research studies the symptoms, root causes, impact, triggers, fixes, and system dependency of bugs in the Puppet configuration management system. Puppet is a widely used open-source configuration management system that performs various administrative tasks on machines based on a central specification. This paper aims to fill the research gap and help improve the understanding of these systems. 2146 registered bugs and fixes are collected and a random sample of 100 is analyzed to answer the research questions. The most common bug types have the symptom of unexpected runtime behavior. The root cause is generally incorrect target machine operations. The impact severity is typically medium and the impact consequence of most bugs is target configuration failure. Bug fixes tend to be small method code changes and usually occur in the execution component. Bugs are generally system-independent and are caused by logic errors that can be triggered by specifically executing an erroneous module. The analysis shows that applying automated bug discovery methods to the most error-prone execution and parser components of the system would lead to the largest improvement in the system’s quality.

Configuration management systems are a class of software used to automate system administrative tasks, one of which is the configuration of software systems. Although the automation is less error-prone than manual configuration done by a human, bugs in the source code can still cause configuration errors. This can result into unwanted consequences for the managed software system, some examples are performance degradation, downtime and security breaches.

Bug studies are conducted on previously reported bugs to find out their characteristics. Such findings help with preventing, detecting and fixing bugs, which ultimately causes software to become more reliable. This paper aims to fill in a research gap in the literature surrounding bug studies conducted on configuration management systems. From Salt, a widely used open-source configuration management system, a data-set of 5,896 bugs with fixes was collected. The main research question answered by this paper is ``What are the common patterns can be extracted from bugs found in Salt?'' To answer this, 100 bugs were randomly sampled from this data-set to analyze their symptoms, root causes, impact, fixes, system-dependence and triggers.

The study of bugs can provide important information to understand their nature in the context of complex software systems as well as supporting developers in their detection, fix and prevention. Previous studies focused on analyzing bugs under different perspectives such as changes at code level, frequency, semantics, symptoms, root causes and reproducibility through test cases. Although these studies offer valid methodologies that can be applied in different areas of bugs analysis, literature suggests that very little focus has been aimed towards configuration management systems.

This research has the goal to provide a characterization of bugs in the Moby configuration management system, an open framework used to create container systems. A random sample of 100 collected Moby bugs is manually inspected and categorized by their (1) symptoms, (2) root causes, (3) impact, (4) fixes, (5) system dependency and (6) triggers. Some representative takeaways suggest that: bugs symptoms are mostly linked to a specific root cause, which means that bugs occur in certain areas of the system; the modular nature of Moby drives up the criticality of the bugs as each component needs to work correctly; bugs are overall hard to reproduce as only 24.5\% of the fixes included a test case; developing an automatic tool that provides a historical distribution of bugs can support maintainers in their work by enhancing bugs prevention.

The research provides an analysis and categorization of bugs in the Moby configuration management system. The work adds a new perspective to the literature on the topics of both bugs analysis and configuration management systems and can be used as a starting point for further studies.

Realistic vehicle routing problems have been highly relevant for years in a wide variety of domains. One such domain is food delivery, where well-crafted routes can reduce costs and contribute to customer satisfaction. This thesis formulates a problem variant for the restaurant meal delivery problem in order to examine the reoptimization of meal delivery routes. A novel solution algorithm that combines rule-based reinforcement learning and adaptive large neighbourhood search is used to tackle the problem. This hybrid algorithm manages to incorporate both learning and handcrafted search heuristics, as well as both offline and online computation. Analysis of the algorithmic components shows that the proposed algorithmic approach is computationally feasible for small scale reoptimization problems. Analysis also demonstrates that the reinforcement learning component currently does not improve performance with respect to the established objective function.