The prevalence of partial differential equations (PDEs) in modeling physics and the low speeds of numerical solvers demands more efficient solving methods. For this purpose, machine learning based methods have been proposed, but these are typically discrete, difficult to interpret and require large amounts of ground truth data to train. To address these issues, we propose a novel machine learning based solver that builds upon advancements in Gaussian based reconstruction. Our method represents the solution to a time-dependent target PDE purely in terms of Gaussians, making it completely continuous and meshless. These Gaussians are evolved by means of an autoregressive neural network that is applied to each Gaussian, integrating information from a local neighborhood of Gaussians. This enables the change of position, scale, rotation and value of the Gaussians to model the solution in a particle-like manner. Extensive experiments show the potential for Gaussians to model arbitrary physical phenomena. We also compare our approach to ground truth data and various state of the art methods, which demonstrates that the method performs well on short-term prediction, but does not match the state of the art for long-term prediction.

Multi-view image recognition is crucial for numerous applications such as autonomous vehicles and robotics, where accurate 3D reconstructions from 2D images are essential. However, the presence of various noise factors like motion blur, variable lighting, and changes in the field of view can significantly degrade the quality of these reconstructions. Through a series of experiments using simulated data created in Blender, I explore the effectiveness of different de-noising methods, including AI-based and traditional image processing techniques. My findings indicate that specific adjustments in the Gaussian Splatting data preprocessing can significantly mitigate the adverse effects of noise, leading to more accurate and reliable 3D reconstructions. The study contributes to the field by providing a detailed analysis of noise impact and proposing viable solutions to enhance the fidelity of multi-view image reconstructions in noisy environments. The code and resources developed for this project are available in the GS-preprocessor repository on GitHub. https://github.com/surftijmen/GS-preprocessor.

Convolutional Neural Networks (CNNs) benefit from fine-grained details in high-resolution images, but these images are not always easily available as data collection can be expensive or time-consuming. Transfer learning pre-trains models on data from a related domain before fine-tuning on the main domain, and is a common strategy to deal with limited data. However, transfer learning requires a similar domain with enough available data to exist, and transferability varies from task to task. To deal with limited high-resolution data we propose resolution transfer: using low-resolution data to improve high-resolution accuracy. For resolution transfer, we use Continuous kernel CNNs (CKCNNs) that can adapt their kernel size to changes in resolution and perform well on unseen resolutions. Training CKCNNs on high-resolution images is currently significantly slower than CNNs. We lower the inference costs of CKCNNs to enable training on high-resolution data. We introduce a CKCNN parameterization that constrains the frequencies of kernels to avoid distortions when the kernel size is changed, improving resolution transfer accuracy. We improve fine-tuning with a High-Frequency Adaptation module that complements our constrained kernels. We demonstrate that CKCNNs with kernel resolution adaptation outperform CNNs for resolution transfer tasks with no fine-tuning or with limited fine-tuning data. We compare to transfer learning, and achieve competitive classification accuracy with an ImageNet pre-trained ResNet-18. Our method provides an alternative to transfer learning that uses low-resolution data to improve classification accuracy when high-resolution data is limited.

Radiance fields are a promising alternative to conventional 3D representations in the domain of novel view synthesis, with recent research achieving truly impressive photorealistic view synthesis results. In this paper, we deal with the concept of non-photorealistic rendering in the context of radiance fields, for generating more stylistic captures of real-world objects. We discuss the suitability of inverse rendering as a stepping-stone to traditional shading and edge detection, and contribute a new algorithm specifically for outline detection in radiance fields.

Neural radiance fields (NeRF) based solutions for novel view synthesis can achieve state of the art results. Recent work proposes models that take less time to render, need less training data or take up less space. However, few papers explore the use of NeRFs in classic rendering scenarios such as rasterization, which could contribute to wider adoption. Our paper tackles the issue of shadow generation and proposes a deep residual MLP network with fast evaluation times, that generates view-dependent shadow maps. The network distills the knowledge of an existing NeRF model and achieves the speedup through the use of neural light fields, by only doing one network forward per ray.

With the current state-of-the-art research, exporting a NeRF to a mesh has the side effect of having to evaluate a Multi Layer Perceptron at render-time, causing a significant decrease in performance. We have found a way to use K-Means clustering to pre-compute values for this MLP, storing them in multiple octahedron maps for the GPU to fetch when it's time to render the object. This improves render times by a factor of 3-4x.

Neural Radiance Fields (NeRF) and their adaptations are known to be computationally intensive during both the training and the evaluating stages. Despite being the end goal, directly rendering a full-resolution representation of the scene is not necessary and not very practical for scenarios like streamed applications. Our goal is to design a streamable adaptation for a model that can produce fast, rough estimates of 3D scenes, by only using a shallow part of the network. The quality is subsequently improved as more parts of the network are available, such that it can be used in online applications where the model needs to be transferred. Separate models can be trained at different resolutions, but this approach results in a large space overhead and also increases the evaluation time. This can be mitigated by reducing the depth of low-resolution models, but redundancy will still be high as each new model needs to re-evaluate the input data, rendering previous calculations obsolete. Our method combines key concepts from previous approaches to create a progressively trained model that is able to produce intermediate outputs of increasing quality while attempting to optimize the trade-off between overhead and quality. Our model is able to produce a recognizable representation of the scene with as little as one hidden layer from the original model. It also allows for division into streamable chunks which can be sent individually and, upon reconstruction, provide intermediate outputs that bring consistent improvement in quality. The newly streamed data uses the residual output from previous computations in order to reduce redundancy. We show that the final quality of our adaptation is within 2% of the original in terms of previously used quantitative metrics.

Generating synthetic images has wide applications in several fields such as creating datasets for machine learning or using these images to investigate the behaviour of machine learning models. An essential requirement when generating images is to control aspects such as the entities or objects in the image. Controlling this helps in creating custom datasets tailored for the above applications and creates a platform to conduct diverse experiments with the generated images, enabling research in the application's field.Existing methods individually enable controllability over various elements in the image such as selecting the objects, their properties, colour or the relations between objects, etc. but we identify a research gap in this field where no single method allows the user to control all these aspects of the image. An additional research gap identified is that existing methods cannot generate images based on a query with a logic based combination of entities.

In this thesis, we aim to fill this research gap by developing SceneUI - a system that allows the user to specify and control aspects of the scene through a user interface such as the objects, object properties, spatial relations between objects, object colour, extent of contextual objects and the background of the image. Additionally, we include a component where the scene is generated based on an OR query specifying the objects as predicates, which serves as a foundation to generating images based on entity combinations. Owing to the limited range of attributes for objects and the lack of objects with attributes in the dataset, we augment the dataset by expanding the attributes of objects in the scene graphs of the dataset and introducing additional objects that have attributes. This was done by identifying recurring objects in the dataset that could be expanded and manually annotating the changes in the dataset. This increases the level of controllability and gives a wider range of an object's properties to choose from.

The goal of the thesis is to design and develop a method that allows the user to declare, specify and manipulate elements of the image and eventually use the images generated for two use cases - Generating Images for Interpretable Machine Learning and Generating Images from Queries as Ground Truth. We evaluate SceneUI to show its usability and effectiveness for the two use cases through two experiments. In the first experiment, we use SceneUI to create biased datasets where each bias is based on an object's colour or object type and the goal is to train a deep learning model that learns the biases. The results show that the model learns the biases well and thus, SceneUI can be used to control datasets which can be used to benchmark machine learning explainability methods. The second experiment generates a dataset of images using the OR query and training machine learning models on the dataset to ensure the suitability of the images for machine learning tasks. As the generated images will be used as ground truth given a query for a specialised machine learning model, the model is expected to identify the predicate objects in the image. The results show SceneUI can generate images based on objects in the query and can also accommodate the objects to have properties. The models identify important features in the SceneUI-generated images and are thus suitable to be used as ground truth. We also discuss the limitations and tradeoffs of SceneUI and present potential future directions for research to improve its scalability.

In this work, we look at the intersection of Sustainable Software Engineering and AI engineering known as Green AI. AI computing is rapidly becoming more expensive, calling for a change in design philosophy. We consider both training and inference of neural networks used for image vision; to reveal energy-efficient practices in an exploratory fashion.

First of all, we examine a modern algorithm for hyperparameter optimisation and compare this to two baseline methods. We find that the baseline algorithms perform considerably worse despite their wide usage and argue that they should not be used when training large models. Furthermore, we look at the layer structure of convolutional networks and conclude that the convolutional layers have the largest influence on the total consumption. We report increases of up to 95% with only marginal improvements in accuracy. Therefore we recommend developers to reduce their network architectures as long as the performance stays within a reasonable margin.

Second, we present a study focused on the inference phase of the deep learning pipeline. We look at the effect of batching for image classification requests. To facilitate the data collection, we make use of a simulated queue and the Pytorch framework. We find that batching has a significant impact on the energy consumption, but the magnitude of this impact can vary a lot for different models. Our recommendation is to treat the batch size as an inference parameter that needs to be tuned first. Additionally, we highlight how the energy consumption of image vision networks has evolved over the past decade. Presenting the findings together with the performance of these networks shows a steady, upward energy trend accompanied by a decreasing slope for the accuracy. The only exception is the model ShuffleNetV2. We mention the design principles that went into the development of this network and present it as a start for future research.

The appearance of an object or scene is determined by factors like the material, the lights, the geometry, the position of the observer, and the surroundings.
Changes in these factors can be simulated using a projector-camera setup.
Other research focuses on changing the appearance from the perspective of the projector, or on projector compensation for slightly warped planar surfaces.
This paper aims to simulate changes in the scene's appearance by actively manipulating the lighting using a projector-camera setup.
It works on not only planar surfaces, but also on more complex geometries.
This is achieved by first doing a calibration, and then using this to optimize a projection.
This projection is optimized to minimize the difference between how the scene looks when the projection is projected and the desired scene.
For low-resolution projectors, it can do this in a few seconds to half a minute.
For higher resolutions, the calibration time and file size get quite big.
This can be solved in future work using different calibration methods.

In this paper, a method is proposed for stylizing stereoscopic augmented reality, while attempting to retain temporal and visual coherence. By tracking an AR marker and estimating a disparity map from stereo images, world positions of anchor points are tracked across frames and images, in order to move them with the scene. Additionally, the density of anchor points in the image is adjusted each frame to prevent areas from being over- or underpopulated with anchor points. The method is applied to a simple mosaic style and both quantitative and qualitative results demonstrate improved coherence when compared to a baseline. Besides that, the algorithm runs at near-interactive frame rates.

Diminished reality (DR) is an extension of augmented reality (AR) in which real objects are concealed, removed, or replaced. State of the art DR implementations are written and evaluated on desktop platforms, and those which are aimed at smartphones use auxiliary data or hardware for image completion. This paper introduces a DR app for Android smartphones which can select, track, and remove objects in an unrestricted environment from live video at interactive speed. The application relies exclusively on RGB frames from the phone's camera as input to the DR pipeline. Multiple algorithms for object selection and tracking, as well as a patch-based inpainting approach were adapted. Selection and tracking of objects was very reliable at interactive speed. Inpainting led to convincing results for planar surfaces and unstructured textures, often delivering acceptable results even in more complicated scenes. Future optimizations are required for real-time performance of DR algorithms even on a flagship smartphone, as the hardware was a constraining factor. The performance of inpainting in particular needs to be improved to achieve acceptable frame rates.

Augmented Reality (AR) tracking for mobile devices is not realiable in environments where large virtual content, such as an entire virtual building, is to be displayed to the user. This paper presents a study on how multiple on-site markers can be used to better align the virtual scene to the real-world environment to improve large-scale AR experiences. The method consists of detecting QR markers from the video stream, and then updating the position, orientation, and scale of the virtual content in order to minimize the error between the real markers and their corresponding virtual markers. This allows for more accurate tracking and can be used in more complex environments. Three different evaluation methods are proposed. Additionally, experiments were conducted to explore the influence of marker density and marker layout. These experiments show that higher density layouts as well as structured layouts lead to higher accuracy and stability.

According to the Contrast Sensitivity Function, the contrast of different spatial frequency elements of an image will be seen differently depending on distance. We propose a method for compensating for these differences by acting on an image's frequency representation and manipulating the magnitudes of the individual wave functions composing the image. We provide an open source implementation of the algorithm which can be used to process videos and images in real time.