The ever-increasing complexity of Artificial Intelligence (AI) models has led to environmental challenges due to high computation and energy demands. This thesis explores the application of tensor decomposition methods—CP, Tucker, and TT—to improve the energy efficiency of large Convolutional Neural Networks (CNNs) during inference by reducing energy consumption. The energy consumption of several convolution layers was measured using a watt meter across various CNN configurations and different hardware architectures (Central Processing Unit(CPU) and Graphics Processing Unit (GPU)). In addition, several regression models were fitted to estimate energy savings, incorporating memory usage. It was found that TT decomposition consistently provided the most significant energy savings across various compression ratios, influenced by CNN hyperparameters such as input/output channels, feature sizes, and kernel sizes, whereas CP decomposition was the least effective in reducing energy. The GPU implementations generally resulted in additional energy consumption, and the GPU regression models suggested a need for more complex relationships. The thesis also revealed that the efficiency of tensor decompositions might be highly dependent on the implementation details of software libraries, such as TensorlyTorch, which can significantly impact the computation and memory complexities. These findings underscore the importance of both hardware specific considerations and careful software implementation in achieving energy efficient CNNs, providing a foundation for further research in energy-constrained environments.

Algorithms which can effectively detect epileptic seizures have the potential to improve current treatment methods for people who suffer from epilepsy. The current state-of-the-art methods use neural networks, which are able to learn directly from the electroencephalogram (EEG) data without feature extraction. However, neural networks have drawbacks—they are time and data intensive to train and require a high number of parameters for efficient classification. This thesis proposes a novel approach to the epileptic seizure detection problem using Support Tensor Machines (STMs). The final models learn directly from EEG data and use far fewer model parameters compared to a state-of-the-art model using a convolutional neural network. Three types of experiments have been conducted using different representations for the EEG data. The STMs used in the experiments are the Support Higher-Order Tensor Machine, Dual Structure-preserving Kernel (DuSK), and Tensor Train Multi-way Multi-level Kernel (TT-MMK). The results show that by using TT-MMK (with a tensorized data representation) for leave-one-seizure-out validation and DuSK (with the original data representation) for leave-one-patient-out validation, the models are able to rival a state-of-the-art epileptic seizure detector.

Compressed sensing is a framework in signal processing that enables the efficient acquisition and reconstruction of sparse signals. A widely-used class of algorithms that are used for this reconstruction, called greedy-algorithms, depend on non-convex optimization. With increasing signal size, these problems become computationally very hard. Block compressed sensing is a framework in compressed sensing that divides the compressed sensing problem into sub-problems, gaining a better storage complexity. However, block compressed sensing has not yet been studied form a computational complexity perspective.

This thesis focuses on the application of block compressed sensing to signals of high dimension to gain insight into the relation between reconstruction performance and computational complexity. This is done by, first investigating how theoretical reconstruction guarantees change, when the problem is divided into smaller sub-problems and by doing a complexity analysis of the reconstruction itself. Each sub-problem solves for a portion of a signal, defined as a block. Next, experiments are conducted in order to get insight into the trade-off between computational complexity and quality of the reconstruction. It can be found that, by using this block-wise approach, the computational complexity of the reconstruction problem decreases, but at the same time, quality of the reconstruction deteriorates. Besides, a method to compensate for this performance loss is proposed. The key idea of this method is that, by propagating prior information among the different blocks, the reconstructions of the blocks can be improved. Finally, block compressed sensing and prior-aware block compressed sensing are analysed in a higher order tensor compressed sensing setting. Nevertheless, this setting was found to exhibit a less favourable complexity-performance trade-off than the normal one, as this setting resulted in both a more complex and a less accurate reconstruction than the normal one.

This thesis applies the Gauss-Newton optimizer to estimate the parameter values of the Volterra-PARAFAC model by minimizing a nonlinear least square cost (NLS) function given the input and output measurements of the MISO Volterra system.

Natural Language Processing (NLP) deals with understanding and processing human text by any computer software. There are several network architectures in the fields of deep learning and artificial intelligence that are used for NLP. Deep learning techniques like recurrent neural networks and feed-forward neural networks are used to develop language models that perform several NLP tasks. Over the years, researchers have worked on developing state-of-the-art language models that achieve high accuracy and performance for NLP applications. With the development of deep neural network language models, the computational resources requirements and the energy costs for training and running language models increased. This led to research to compress the language models, thereby reducing the computational complexity of the language models. One of the methods used for this is tensor decomposition, like the tensor-train (TT) decomposition. During this thesis work, the application of the TT-decomposition method for compressing the embedding layer in a long-short-term memory model was investigated. Specifically, the effect of factorization and the order of factors in the embedding layer when it is represented in the TT-matrix format on the maximum test accuracy of the long- short term memory model for the NLP task of sentiment analysis was investigated. This was done by considering three different factorizations of the embedding layer in the model. Further, the effect of change in TT-ranks (hyperparameters of the model when the embedding layer is represented in the TT-matrix format) on the maximum test accuracy was also investigated. Based on the investigation and empirical results obtained, this thesis concludes that by having a larger number of factors in the factorization of the embedding layer, the maximum test accuracy of the model increases. Further, in a particular factorization, when the factors were arranged in such a way that the maximum values of the TT-ranks had a smaller gap, the maximum test accuracy of the model improved. In one particular configuration of the model, the number of parameters was reduced by 24.5 times compared to that of the original uncompressed model, and a maximum test accuracy of 77.10% was achieved compared to a maximum test accuracy of 78.05% in the case of the original model.

Humans make decisions when presented with choices based on influences. The Internet today presents people with abundant choices to choose from. Recommending choices with an emphasis on people's preferences has become increasingly sought. Grundy (1979), the first computer librarian Recommender System (RS), provided users with book recommendations. Growing volumes of user data in the '90s saw increased usage of commercially available RS for e-commerce, music, movies, books, and social networking services. Due to their effectiveness in providing recommendations, Collaborative Filtering (CF) algorithms are predominantly used to build these RS. However, traditional CF algorithms adopting Matrix Factorization (MF) and Nearest Neighbor (NN) methods suffer from handling sparse data or model scalability. With exponentially increasing sparse data, building scalable and accurate RS models is of focus.

This thesis uses tensors and graphs to represent available data. Emphasis is given to capturing higher-order interactions present between the data. The use of tensors is motivated as matrices cannot capture data with higher-order relations, such as variation of user ratings to items with time. The transition to using tensors has been promising with the development of efficient tensor decomposition methods and powerful machines. Graphs can capture the correlation between different entities, providing additional information intrinsic to the underlying graph structure. A Graph Regularized CANDECOMP/PARAFAC (GRCP) tensor decomposition model framework is proposed in this thesis. The thesis highlights how to graph Laplacian regularizers (GLRs) benefit CP tensor decomposition methods to build RS. The model framework is evaluated with the MovieLens data set. The model records lesser Normalized Mean Squared Error (NMSE) values than those reported in the literature. The combination of varied data sources notably aids in overcoming the drawbacks of current RS models, offering scalability with computational efficiency in linear time.

This thesis studies the Canonical Polyadic Decomposition (CPD) constrained kernel machine for large scale learning, i.e. learning with a large number of samples. The kernel machine optimization problem is solved in the primal space, such that the complexity of the problem scales linearly in the number of samples as opposed to scaling cubically in the dual space. Product feature maps are applied to transform the input data. The weights are constrained to be a CPD, so the number of weights scales linearly in the number of features. The CPD introduces a nonlinearity, so nonlinear optimization must be applied.
It is studied in which situation it is more advantageous to apply iterative all-at-once opti- mization compared to Alternating Least Squares (ALS) to solve the CPD constrained kernel machine problem. Specifically, all-at-once gradient descent methods are studied. An efficient analytical algorithm for the all-at-once gradient is derived. Furthermore, it is shown that automatic differentiation (AD) can also be applied, but it is found to be slower than the analytical method.
The selection of a step size is found to be challenging. It is shown that the magnitude of the gradient of the mean squared error (MSE) term decreases for an increasing number of features. As a result, selecting the step size becomes more difficult for more features. To overcome this, the Line search and the Adam method are studied. A general expression for the exact line search solution is derived. It can be applied to compute the optimal step size for any step direction and any number of features. However, the Adam method performs better in terms of loss after training, convergence and the training run time. The mini-batch Adam method is used to evaluate the performance of all-at-once optimization for large scale learning.
It is found that the Adam method no longer performs well for data sets with around 16 features or more, likely due to the decrease in the magnitude of the gradient of the MSE term. On large scale data sets with fewer features, the Adam method outperforms ALS in terms of run time until convergence while achieving similar training and validation losses. The Adam method reached convergence on a data set with 11 million samples within ten minutes. Furthermore, it is shown that the scaling of the run time of the Adam method in terms of the feature map order and the CP-rank is more than an order lower than the scaling of ALS when the methods are run on a GPU. This makes to Adam method more suitable for more complex models.

Online video completion aims to complete corrupted frames of a video in an online fashion. Consider a surveillance camera that suddenly outputs corrupted data, where up to 95% of the pixels per frame are corrupted. Real time video completion and correction is often desirable in such scenarios. Therefore, this thesis improves the Tensor-Networked Kalman Filter (TNKF) as presented in [12] by developing the Tensor-Networked Square-Root Kalman Filter (TNSRKF). The TNSRKF is a Square-Root Kalman Filter (SRKF) realized in a Tensor Networks (TN) structure. The square-root Kalman filter is an inherently stable algorithm, and indirectly allows for more information retention throughout the algorithm. Thereby, the filter aims to improve the performance of the TNKF. Furthermore, implementing the filter in TN results in an intuitive implementation of the filter while allowing for larger video dimensions than the widely used matrix format. This thesis concludes that the TNSRKF is too computationally burdensome to complete a video in an online fashion due to the implementation of the Modified Gram-Schmidt (MGS) algorithm in Tensor-Train (TT)-format. In addition, results demonstrate that a low-rank orthogonality matrix is not realizable, making it impractical to update the state covariance matrix via any method that uses an orthonormal matrix.

Nowadays, video surveillance and motion detection system are widely used in various environments. With the relatively low-price cameras and highly automated monitoring system, video and image analysis on road, highway and skies becomes realistic. The key process in the analysis is to separate the useful information such as moving foreground objects from the original video sequence where Robust Principal Component Analysis (RPCA) plays an important role in extracting the foreground objects. RPCA have been widely used in data analysis and dimension reduction with applications in image recovery, information clustering and computer vision. But one drawback of RPCA lies in the fact that it does not guarantee the nonnegativity of pixels. It is important to have nonnegative foreground object since negative pixels that are not in the range between 0 and 255 are meaningless and the foreground objects are thus not visible. State-of-the-art methods do not consider the nonnegativity of the foreground object in their algorithms. This thesis focuses on the problem of extracting foreground moving object from background scenes and guarantee the nonnegativity of foreground object. This thesis proposes a method that combines RPCA and Nonnegative Matrix Factorization (NMF). It ensures the pixels that constitute the foreground object is nonnegative by using the basic model of RPCA and nonnegative components that NMF provides. The efficacy of the proposed algorithms is tested on publicly available dataset. Experiment shows in detail how the proposed algorithms achieve in recovering the foreground object with high true positive rate. Together with RPCA algorithm, the performance of recovery is compared and their advantages and disadvantages are discussed.

Least-squares support-vector-machines are a frequently used supervised learning method for nonlinear regression and classification. The method can be implemented by solving either its primal problem or dual problem. In the dual problem a linear system needs to be solved, yet for large-scale problems this can be impractical as current methods suffer from the \textit{curse of dimensionality}. This phenomena causes the computational and memory requirements to exceed the capabilities of standard computers for large datasets. In this thesis, a tensor network Bayesian learning method was developed to avoid these burdensome complexities. The developed method performs competitively with the current state-of-the-art, and unlike other low-rank approximation methods, allows for incorporation of user-knowledge, noise, early stopping, and yields confidence bounds on the obtained model.

B-splines are basis functions for the spline function space and are extensively used in applications requiring function approximation. The generalization of B-splines to multiple dimensions is done through tensor products of their univariate basis functions. The number of basis functions and weights that define a multivariate B-spline surface, therefore, increase exponentially with the number of dimensions, i.e. B-splines suffer from the curse of dimensionality. Tensor network theory provides a mathematical framework to alleviate the curse of dimensionality of B-splines by representing the high-dimensional weight tensor as a low-rank approximation. This thesis presents the Tensor Network B-spline (TNBS) model, along with an optimization algorithm that allows the estimation of the exponentially large weight tensor directly from data, without ever needing to explicitly construct it. P-spline regularization is incorporated to induce additional smoothness and ensure the B-spline hypersurface generalizes well across the high-volume domain. The developed TNBS framework opens doors for the application of B-spline theory in high-dimensional function approximation. This thesis provides an overview of both B-spline and tensor network theory, then uses it to derive the TNBS model. We validate the effectiveness of the model through an application in black-box nonlinear system identification using a NARX approach. An open-source MATLAB implementation of TNBS is made available on GitHub. The work is concluded with some recommendations for further research on this topic.

One of the most promising ideas in autonomous vehicle control systems is letting the vehicle drive autonomously outside the normal, linear, operating region and letting it "drift". By doing so, the maneuverability of the vehicle could be enhanced. To enable systems that can control this behaviour, estimation of certain vehicle states is needed with high accuracy and high frequency.In this project, a new solution to this problem is proposed by combining a mixed dynamic-kinematic observer with a single camera that produces velocity measurements based on tracking the ground plane. To improve filtering of the camera velocity measurements, the measurement error covariance matrix is updated online based on a model of the camera measurement error. Evaluation of the new methodology was done on data recorded from a 1:10 scale test vehicle and performance was assessed based on ground truth data obtained using a Motion Capture System.In normal driving conditions with correctly identified vehicle parameters, an observer without camera performs better by 25% in terms of RMSE on lateral velocity and sideslip angle estimation. However, the online adaptation of the covariance matrix results in an estimate that is at least 45% more accurate in terms of RMSE than the same observer without online covariance adaptation. Next to that, experiments show that the proposed observer with camera has better robustness to uncertainty in model parameters by almost a factor five in terms of RMSE than the observer without camera.When the grip of the tires is physically lowered and the vehicle is drifting, the proposed observer can track large sideslip angles (>30°), where the state-of-the-art observer without camera is not able. The state-of-the-art observer has an increase in RMSE of 75% on all estimated quantities in comparison to the proposed methodology. These results show that adding a camera to an existing sideslip angle observer greatly enhances robustness of the observer to uncertainty in model parameters and violation of model assumptions. This comes dat the cost of losing some accuracy in normal driving conditions.

In the thesis, the challenge of precisely developing a data-driven Linear Time Invariant MIMO Reticle Heating Induced Deformation Prediction (RHIDP) model for ASML's DUV systems is presented. The model is developed for two inputs, namely airflow temperature and dose for full field exposures. A reduced order data-driven based approach for developing a RHIDP model for various Use Cases (UC) is presented. This prediction model will be used for precisely estimating the global reticle deformation geometry as well as remove engendered non-Moving Average (non-MA) effects below the reticle surface. The identified linear regression model exhibits a very high degree of prediction accuracy for a broad working envelope, the prediction is accurate to within a range of 10^(-12) m and 10^(-10) m with 99% VAF values for all static inputs (UCs) and low frequency (<= 0.01 Hz) sine inputs. This model can thus be used for precise feed-forward correction of RH induced thermal deformation and non-MA effects