This paper proposes a sparse Bayesian treatment of deep neural networks (DNNs) for system identification. Although DNNs show impressive approximation ability in various fields, several challenges still exist for system identification problems. First, DNNs are known to be too complex that they can easily overfit the training data. Second, the selection of the input regressors for system identification is nontrivial. Third, uncertainty quantification of the model parameters and predictions are necessary. The proposed Bayesian approach offers a principled way to alleviate the above challenges by marginal likelihood/model evidence approximation and structured group sparsity-inducing priors construction. The identification algorithm is derived as an iterative regularised optimisation procedure that can be solved as efficiently as training typical DNNs. Remarkably, an efficient and recursive Hessian calculation method for each layer of DNNs is developed, turning the intractable training/optimisation process into a tractable one. Furthermore, a practical calculation approach based on the Monte-Carlo integration method is derived to quantify the uncertainty of the parameters and predictions. The effectiveness of the proposed Bayesian approach is demonstrated on several linear and nonlinear system identification benchmarks by achieving good and competitive simulation accuracy. The code to reproduce the experimental results is open-sourced and available online.

@en

Applying deep neural networks (DNNs) for system identification (SYSID) has attracted more andmore attention in recent years. The DNNs, which have universal approximation capabilities for any measurable function, have been successfully implemented in SYSID tasks with typical network structures, e.g., feed-forward neural networks and recurrent neural networks (RNNs). However, DNNs also have limitations. First, DNNs can easily overfit the training data due to the model complexity. Second, DNNs are normally regarded as black-box models, which lack interpretability and cannot be used for white-box modelling. In this thesis, we develop sparse Bayesian deep learning (SBDL) algorithms that can address these limitations in an effectivemanner.@en

Hand anthropometry is one of the fundamentals of ergonomic research and product design. Many studies have been conducted to analyze the hand dimensions among different populations, however, the definitions and the numbers of those dimensions were usually selected based on the experience of the researchers and the available equipment. Few studies explored the importance of each hand dimension regarding the 3D shape of the hand. In this paper, we aim to identify the dominant dimensions that influence the hand shape variability while considering the stability of the measurements in practice. A novel four-step research method was proposed where in the first step, based on literature study, we defined 58 landmarks and 53 dimensions for the exploration. In the second step, 80,000 virtual hand models, each had the associated 53 dimensions, were augmented by changing the weights of Principle Components (PCs) of a statistical shape model (SSM). Deep neural networks (DNNs) were used to establish the inverse relationships from the dimensions to the weight of each PC of the hand SSM. Using the structured sparsity learning method, we identified 21 dominant dimensions that represent 90% of the variance of the hand shape. In the third step, two different manual measuring methods were used to evaluate the stability of the measurements in practice. Finally, we selected 16 dominant dimensions with lower measurement variance by synthesizing the findings in Step 2 and 3. It was concluded that the recognized 21 dominant dimensions can be treated as the reference dimensions for anthropometric study and using the selected 16 dominant dimensions with lower measurement variance, ergonomists are able to generate a 3D hand model based on simple measurement tools with an accuracy of 5.9 mm. Though the accuracy is limited, the efforts are minimum, and the results can be used as an indicator in the early stage of research/design.

@en

One-Shot Neural Architecture Search (NAS) is a promising method to significantly reduce search time without any separate training. It can be treated as a Network Compression problem on the architecture parameters from an overparameterized network. However, there are two issues associated with most one-shot NAS methods. First, dependencies between a node and its predecessors and successors are often disregarded which result in improper treatment over zero operations. Second, architecture parameters pruning based on their magnitude is questionable. In this paper, we employ the classic Bayesian learning approach to alleviate these two issues by modeling architecture parameters using hierarchical automatic relevance determination (HARD) priors. Unlike other NAS methods, we train the overparameterized network for only one epoch then update the architecture. Impressively, this enabled us to find the architecture on CIFAR-10 within only 0.2 GPU days using a single GPU. Competitive performance can be also achieved by transferring to ImageNet. As a byproduct, our approach can be applied directly to compress convolutional neural networks by enforcing structural sparsity which achieves extremely sparse networks without accuracy deterioration.@en