This thesis was written in during my internship at Adyen as the final project of the Master’s program in Computer Science at the TU Delft. In my Bachelor’s thesis, I compared the energy consumption of three Android UI frameworks, and I chose to continue working on the subject of energy consumption on Android as it is relevant both to Adyen and the greater Android community. In this thesis, I review the state-of-the-art of energy consumption tools for Android, compare three approaches to attributing energy consumption to code at a fine-grained level, and implement one of these in a tool that can be used to analyze release-mode Android apps. I then empirically evaluate this tool, and apply it at Adyen in a case study in which I cooperate with a developer to solve an energy bug. As part of the empirical evaluation, I also perform a preliminary comparison of release and debug mode with respect to energy consumption, using three code smells identified in prior work as having an effect on the energy use of methods containing them. The results of the empirical investigation show that statistical sampling can be applied to Android devices to attribute energy consumption to methods within a reasonable margin of error. It further shows that this approach can be used to identify differences in energy consumption between different versions of software. The comparison between release and debug mode showed that the overhead caused by the use of debug mode is not consistent, and varies between both different code smells and different devices. This has significant implications for further work measuring energy consumption on Android, as it implies that results obtained using debug mode cannot be generalized to release mode, which all apps will use in production environments. Finally, the case study showed that this approach is able to significantly assist the energy debugging process, and revealed that even today, developers often lack the information necessary to make informed decisions over the energy consumption of their software. However, once this information is made available, both developers and stakeholders are willing to adapt their decision-making to incorporate energy efficiency.

Neural Radiance Fields (NeRFs) have demonstrated remarkable capabilities in photo-realistic 3D reconstruction. NeRFs often take as input posed images where the camera poses come from either off-the-shelf S\textit{f}M or online optimization together with NeRFs. However, we find that both strategies yield suboptimal results in recovering camera poses from images when encountering texture-less and repetitive patterns, particularly in aircraft engine inspection. To reconstruct photo-realistic 3D engine blades from images, we propose BladeNeRF, a new variant of NeRF model that incorporates camera constraints into learning and enables accurate pose learning. In addition, we propose to separate the blades in the foreground from the constant background, eliminating background artefacts and enhancing depth estimation accuracy. Experimental evaluations on synthetic data demonstrate the advantage of our model in precise camera pose estimation and high-fidelity 3D scene reconstruction compared to other NeRF variants.

With a 5-year survival rate of only 9%, pancreatic cancer is one of the deadliest types of cancer. Among other things, this is caused by the extreme difficulty of diagnosing recurrent pancreatic cancer in an early stage. One of the important next steps in discovering pancreatic cancer automatically on a CT scan is finding healthy pancreatic tissue. This research explores the feasibility of segmenting the pancreas on a CT scan using a deep learning approach, focusing on postoperative cases after pancre- atic resection, by combining state-of-the-art segmentation models for the preoperative pancreas with novel techniques, such as 3D Multi-Scale Convolutional Blocks (MCBs) and KNet architectures. Us- ing pretraining on preoperative data, a complete segmentation pipeline was designed to segment the pancreas in the postoperative state. The experimental results demonstrate that deep learning models can effectively segment the pancreas, despite the increased complexity and difficulty of the postoper- ative state. Notably, employing advanced techniques such as the KNet architecture and MCBs yields significant performance improvements. The newly designed architectures presented in this research, MKNet, MSKNet, and MAKNet, achieve state-of-the-art results for both preoperative and postoperative pancreas segmentation, greatly improving the Hausdorff Distance (HD) and 95th Percentile Hausdorff Distance (HD95) in particular. In the preoperative state, substantial performance improvements were observed compared to the previous state-of-the-art, with a 12.17% increase for HD and a 14.64% in- crease for HD95. Similarly, in the postoperative state, improvements of 11.99% for HD and 13.25% for HD95 were measured. Additionally, qualitative evaluations were conducted with seven experienced radiologists and radiotherapists to validate the algorithm’s performance in a clinical context. Remark- ably, in 83% of the cases, the algorithm was evaluated to accurately segment the pancreas, requiring minimal or no modifications according to the medical experts. This study not only contributes to the state-of-the-art in pancreas segmentation but also introduces comprehensive quantitative and qualitative evaluation methodologies. Furthermore, it addresses the segmentation of the postoperative pancreas, which has not yet been touched upon in previous re- search. Limitations are acknowledged, such as the availability of a limited dataset, and future research directions are proposed, including generalizability and enhancements to the proposed models. Despite the limited availability of data, which impacts the performance and generalization capabilities to some degree, the research in this thesis showcases the capability of the MKNet family architectures to ac- curately segment the postoperative pancreas, offering potential benefits for medical applications, data annotation acceleration, and future research in this domain.

Humans interact more and more at a distance these days. Where currently, this interaction is mainly visual and auditory, Tactile Internet (TI) also allows for remote kinesthetic interactions. With the devel- opment of TI, a paradigm shift is on the horizon where the current internet evolves from content delivery to a skill-set delivery network. Imagine the possibilities when specialists in whatever field can, in real- time, apply their skills everywhere in the world from within their living room. To make this a reality, TI applications have to be able to provide feedback to the user with ULL. The speed of light, however, constrains the extent to which TI can be used at large distances. This thesis hence takes a closer look at the problems that arise when long-distance TI is to be used and tries to tackle them. These problems are tackled by building a system based on Model-Mediated Teleoperation (MMT), where lo- cal simulations of the environments are used. A local feedback loop is instantiated through these local simulations, which lowers the system’s dependence on the network performance. The former control loop, including the network, now becomes a corrective loop to prevent mismatches. This thesis focuses on the part where the local simulation is corrected by the real-life data received over the network. The question is how real-time dynamic environments can be captured and kept track of for use in MMT. We identify accurate tracking of objects in the environment as the critical building block enabling MMT with dynamic objects. Consequently, the first 6 Degrees of Freedom (DoF) tracking application designed for use in TI applications is created and evaluated. Through this research, the first steps are set towards a full-blown MMT system that can capture the remote environment successfully.

In this paper, the unintended consequences, also named edge cases in this paper, of integrating fairness into the automated negotiation process are researched. By finding these unintended consequences, we can deal with them accordingly or avoid them, as to not cause any problems with our fairness metric that might make our negotiation process less fair or cause any undesired behaviour. Edge cases are searched for in a small-scale experiment by implementing the difference principle from the 'Justice as Fairness' notion by John Rawls into the negotiation process. The negotiation has two agents, and the behaviour of one of the agents is changed to adhere to the difference principle. By using these agents in automated negotiations with different domains, the behaviour and outcomes of these negotiations between two agents will be checked for any abnormalities that could be considered an unintended consequence. From this, we conclude that agent implementing fairness has a smaller available bidding space which leads to a staler negotiation process. Furthermore, the outcomes show that not always an optimal result is found. However, no unintended consequences directly related to fairness were found. Since finding edge cases is an exhaustive process, which can be compared to finding bugs in a computer program, the research done is not proof that the Rawlsian notion of fairness, or any kind of fairness for that matter, has no other unintended consequences. The research in this paper can be used as inspiration for further research into the edge cases of fairness in automated negotiation, and to have a general idea as to what unintended consequences Rawlsian fairness brings.

Is there a way to incorporate fairness in the opponent modeling component of an automated agent? Since opponent modeling plays an important role in a negotiation strategy, it is reasonable to research how fairness can be integrated into this component, as it influences the outcome of the negotiation. A first step towards finding an answer to this question is to define fairness and how this definition can be translated to algorithmic fairness. The next step is to investigate already available opponent models and assess whether their strategy can be considered fair or not. This paper analyses the process of Bayesian learning in the context of opponent modeling, and tries to reveal possible flaws or strengths that the model has embedded in it, with the hope to add relevant information in the area of automated negotiation.

As automated negotiating agents become more and more part of our daily life, additional care needs to be taken that the agents can negotiate fairly. Humans each have their own intrinsic view on fairness, which affects the negotiation processes and the degree to which the outcome is viewed as satisfactory. However, most current agents are built around a specific notion of fairness, which could be potentially unwanted in certain scenarios or by some individuals. This paper presents a generic framework through which varying fairness notions can be implemented. The performance of the framework is tested through a selection of proposed agents with different fairness views. The results show that the agents are able to reach an agreement on the pareto frontier that is fair given their notion. Thus they do not sacrifice optimality for fairness.

This paper aims to define the broad concept of fairness and investigate how it can be measured, especially considering fairness in automated negotiations. The report relies on the work on fairness issues that have been derived from the research of C. Albin [1]. Firstly, the paper elaborates on different fairness metrics from the literature review. Then these metrics are tested to assess if they capture the effect of considering fairness by the agents in the negotiations. That is done by simulating multiple bilateral negotiations under the open-source GeniusWeb framework, where eccentric agents are compared with fairness-oriented parties by using the fairness metrics. Based on the conducted experiment, the most consistent fairness metric in automated negotiations is the distance to the game-theoretic solutions to the bargaining problem, which considers much of the outcome fairness concept. However, other investigated metrics also capture the different scope of fairness and can be used as metrics, especially when combined, and some interdependence between metrics is modelled.

The field of automated negotiation promises to improve negotiations, thus, a fair outcome and process should also be considered when building these systems. However, issues exist with computational approaches to fairness with which the field of computer science is mainly concerned. To this end, we propose a new approach to fairness based on that of essentially contested concepts to see if argumentation-based negotiation could be used as an extension to the Stacked Alternating Offers Protocol to improve fairness. Looking at fairness as an essentially contested concept shows that discussion between people somehow influenced by the negotiation system is necessary to maintain its fairness. This in turn means that systems that provide accessible context are fairer than systems that would not do so. Thus arguments, if implemented in an accessible manner, add more context to the negotiation, in turn making an SAOP negotiation fairer.

Diminished reality (DR) is an extension of augmented reality (AR) in which real objects are concealed, removed, or replaced. State of the art DR implementations are written and evaluated on desktop platforms, and those which are aimed at smartphones use auxiliary data or hardware for image completion. This paper introduces a DR app for Android smartphones which can select, track, and remove objects in an unrestricted environment from live video at interactive speed. The application relies exclusively on RGB frames from the phone's camera as input to the DR pipeline. Multiple algorithms for object selection and tracking, as well as a patch-based inpainting approach were adapted. Selection and tracking of objects was very reliable at interactive speed. Inpainting led to convincing results for planar surfaces and unstructured textures, often delivering acceptable results even in more complicated scenes. Future optimizations are required for real-time performance of DR algorithms even on a flagship smartphone, as the hardware was a constraining factor. The performance of inpainting in particular needs to be improved to achieve acceptable frame rates.

In this paper, a method is proposed for stylizing stereoscopic augmented reality, while attempting to retain temporal and visual coherence. By tracking an AR marker and estimating a disparity map from stereo images, world positions of anchor points are tracked across frames and images, in order to move them with the scene. Additionally, the density of anchor points in the image is adjusted each frame to prevent areas from being over- or underpopulated with anchor points. The method is applied to a simple mosaic style and both quantitative and qualitative results demonstrate improved coherence when compared to a baseline. Besides that, the algorithm runs at near-interactive frame rates.

Augmented Reality (AR) tracking for mobile devices is not realiable in environments where large virtual content, such as an entire virtual building, is to be displayed to the user. This paper presents a study on how multiple on-site markers can be used to better align the virtual scene to the real-world environment to improve large-scale AR experiences. The method consists of detecting QR markers from the video stream, and then updating the position, orientation, and scale of the virtual content in order to minimize the error between the real markers and their corresponding virtual markers. This allows for more accurate tracking and can be used in more complex environments. Three different evaluation methods are proposed. Additionally, experiments were conducted to explore the influence of marker density and marker layout. These experiments show that higher density layouts as well as structured layouts lead to higher accuracy and stability.

The appearance of an object or scene is determined by factors like the material, the lights, the geometry, the position of the observer, and the surroundings.
Changes in these factors can be simulated using a projector-camera setup.
Other research focuses on changing the appearance from the perspective of the projector, or on projector compensation for slightly warped planar surfaces.
This paper aims to simulate changes in the scene's appearance by actively manipulating the lighting using a projector-camera setup.
It works on not only planar surfaces, but also on more complex geometries.
This is achieved by first doing a calibration, and then using this to optimize a projection.
This projection is optimized to minimize the difference between how the scene looks when the projection is projected and the desired scene.
For low-resolution projectors, it can do this in a few seconds to half a minute.
For higher resolutions, the calibration time and file size get quite big.
This can be solved in future work using different calibration methods.

According to the Contrast Sensitivity Function, the contrast of different spatial frequency elements of an image will be seen differently depending on distance. We propose a method for compensating for these differences by acting on an image's frequency representation and manipulating the magnitudes of the individual wave functions composing the image. We provide an open source implementation of the algorithm which can be used to process videos and images in real time.

In this work, we look at the intersection of Sustainable Software Engineering and AI engineering known as Green AI. AI computing is rapidly becoming more expensive, calling for a change in design philosophy. We consider both training and inference of neural networks used for image vision; to reveal energy-efficient practices in an exploratory fashion.

First of all, we examine a modern algorithm for hyperparameter optimisation and compare this to two baseline methods. We find that the baseline algorithms perform considerably worse despite their wide usage and argue that they should not be used when training large models. Furthermore, we look at the layer structure of convolutional networks and conclude that the convolutional layers have the largest influence on the total consumption. We report increases of up to 95% with only marginal improvements in accuracy. Therefore we recommend developers to reduce their network architectures as long as the performance stays within a reasonable margin.

Second, we present a study focused on the inference phase of the deep learning pipeline. We look at the effect of batching for image classification requests. To facilitate the data collection, we make use of a simulated queue and the Pytorch framework. We find that batching has a significant impact on the energy consumption, but the magnitude of this impact can vary a lot for different models. Our recommendation is to treat the batch size as an inference parameter that needs to be tuned first. Additionally, we highlight how the energy consumption of image vision networks has evolved over the past decade. Presenting the findings together with the performance of these networks shows a steady, upward energy trend accompanied by a decreasing slope for the accuracy. The only exception is the model ShuffleNetV2. We mention the design principles that went into the development of this network and present it as a start for future research.

Generating synthetic images has wide applications in several fields such as creating datasets for machine learning or using these images to investigate the behaviour of machine learning models. An essential requirement when generating images is to control aspects such as the entities or objects in the image. Controlling this helps in creating custom datasets tailored for the above applications and creates a platform to conduct diverse experiments with the generated images, enabling research in the application's field.Existing methods individually enable controllability over various elements in the image such as selecting the objects, their properties, colour or the relations between objects, etc. but we identify a research gap in this field where no single method allows the user to control all these aspects of the image. An additional research gap identified is that existing methods cannot generate images based on a query with a logic based combination of entities.

In this thesis, we aim to fill this research gap by developing SceneUI - a system that allows the user to specify and control aspects of the scene through a user interface such as the objects, object properties, spatial relations between objects, object colour, extent of contextual objects and the background of the image. Additionally, we include a component where the scene is generated based on an OR query specifying the objects as predicates, which serves as a foundation to generating images based on entity combinations. Owing to the limited range of attributes for objects and the lack of objects with attributes in the dataset, we augment the dataset by expanding the attributes of objects in the scene graphs of the dataset and introducing additional objects that have attributes. This was done by identifying recurring objects in the dataset that could be expanded and manually annotating the changes in the dataset. This increases the level of controllability and gives a wider range of an object's properties to choose from.

The goal of the thesis is to design and develop a method that allows the user to declare, specify and manipulate elements of the image and eventually use the images generated for two use cases - Generating Images for Interpretable Machine Learning and Generating Images from Queries as Ground Truth. We evaluate SceneUI to show its usability and effectiveness for the two use cases through two experiments. In the first experiment, we use SceneUI to create biased datasets where each bias is based on an object's colour or object type and the goal is to train a deep learning model that learns the biases. The results show that the model learns the biases well and thus, SceneUI can be used to control datasets which can be used to benchmark machine learning explainability methods. The second experiment generates a dataset of images using the OR query and training machine learning models on the dataset to ensure the suitability of the images for machine learning tasks. As the generated images will be used as ground truth given a query for a specialised machine learning model, the model is expected to identify the predicate objects in the image. The results show SceneUI can generate images based on objects in the query and can also accommodate the objects to have properties. The models identify important features in the SceneUI-generated images and are thus suitable to be used as ground truth. We also discuss the limitations and tradeoffs of SceneUI and present potential future directions for research to improve its scalability.

In the field of computer vision, the difficulty of processing the illumination in an arbitrary scene creates complications for a machine's understanding of the scene. Computer vision algorithms commonly use input images acquired by a camera, where the only measured quantity is the amount of light the camera's sensor receives without any additional information about the shadows, occlusion, highlights, light directionality and interreflections, to help with scene processing and understanding. This thesis applies polarization state measurements of the camera's incoming light to implicitly extract information on these illumination phenomena through a polarization-based intrinsic image decomposition. By defining and measuring the polarization state parameters of an image in the form of polarization cosines, the diffuse and specular intrinsic images can be estimated through an iterative local optimization algorithm to give additional information on the scene illumination. The decomposition algorithm is qualitatively evaluated on three image sample sets, which show that the algorithm performs well as long as the measurements are aligned. The diffuse and specular intrinsic images can then be used to improve existing scene post-processing operations without requiring additional illumination information. One of the three image sample sets is used to evaluate the effects of the decomposition on texture modification, glossiness remapping and tonemapping. The results indicate that the three operations can be either simplified or can have their quality improved with the additional information supplied by the diffuse and specular intrinsic images.