In this thesis, we study the sequential Monte Carlo method for training neural networks in the context of time series forecasting. Sequential Monte Carlo can be particularly useful in problems in which the data is sequential, noisy and non-stationary. We compare this algorithm against a gradient-based method known as stochastic gradient descent (SGD), a commonly used method for training neural networks. The performance of SGD on forecasting non-stationary, noisy time series can be poor due to the possibility of overfitting on the data. The sequential Monte Carlo method may offer a solution for the problems that arise in forecasting non-stationary time series with SGD neural networks. At the same time, neural networks trained with SGD give deterministic predictions, and there is a need for quantification of the uncertainty in the prediction. Sequential Monte Carlo sequentially samples the weights of the neural network, providing a posterior distribution on the weights and thus the outcome. In this work, the sequential Monte Carlo algorithm is tested and analyzed, with different parameter settings, on four time series to give an overview of the behavior. Furthermore, we apply the SMC algorithm on a convolutional neural network known as WaveNet. We show that the SMC algorithm is very well-suited for forecasting non-stationary time series, and can significantly outperform the gradient-based SGD method. Additionally, we show that for specific time series the SMC algorithm on a convolutional neural network outperforms the SMC algorithm on a fully-connected neural network.

Clients with a mortgage loan may prepay a part of their loan before the contractual date. This is called prepayment. In the case of a prepayment, the bank who issued the loan earns less interest than ini- tially agreed. It is therefore essential to build accurate models for predicting prepayment behavior. In this thesis, machine learning models called artificial neural networks are used to predict conditional prepayment rates. In particular, the possibility to quantify the model uncertainty is studied. It is important to acknowledge that the accuracy of a model is not constant over its domain. Most ma- chine learning methods do not provide information about the uncertainty regarding a prediction and, unfortunately, methods that do so are often computationally expensive. This thesis studies uncertainty estimates generated by a neural network with dropout applied. Dropout is a method that randomly drops out neurons in each layer and has been suggested to prevent overfitting. We will see that this method can also be used to extract a measure of uncertainty regarding the prediction of a neural net- work efficiently. This method is used first to evaluate uncertainty for three simple functions under different distributions of the train data. Then the uncertainty estimates are evaluated for a neural network that predicts conditional prepayment rates. It will be illustrated that the uncertainty estimates provide an accurate indication of regions of the domain where predictions are inaccurate. Moreover, using a different model in the regions identified as uncertain can result in higher model performance.