Bayesian Variable Selection in Probability of Default Models

Abstract

Banks are financial institutions that lend money from other parties and provide loans to individuals and organisation for a higher interest. Lending out money is associated with the risk that debtors are not able to fully or partially repay the loans. This is called credit risk. Banks have to make an estimate of the credit risk in their portfolios and have to keep reserves for potential losses. The way this risk is to be determined, is decided by the government where the bank is established. In Europe, the United States, Russia, China among others, the legislation on credit risk is derived from Basel III. Basel III is an international framework to homogenise banking regulation across the world. There are three important factors to determine credit risk In Basel III, namely Probability of Default, Loss Given Default and Exposure at Default. In this thesis I investigate Probability of Default (PD) modelling. The size of the portfolio, for which the Probability of Default has to be estimated, can vary greatly. When the amount of defaults in a portfolio is low and the amount of explanatory variables is high, there is a risk of overfitting. Variable selection methods can be used to counteract overfitting and give understanding of the important predictors. I apply variable selection methods on on a logistic regression. I look at three Frequentist variable selection methods, namely Forward Selection, Lasso and Relaxed Lasso. I compare these three methods with Predictive Projection combined with a Horseshoe prior, which is a Bayesian approach to variable selection. Forward Selection starts with only the intercept in the model and adds variables one by one to the model. The variables are added in such a way that each step increase the estimated performance the most. The Horseshoe prior and Lasso Regression are types of regularisation, where the estimates of the regression coefficients of the logistic regression get shrunk to zero. In Lasso regression, this is done by adding a L1 penalty of the regression coefficients to the logistic regression. This causes weak signals to be pulled to zero. Lasso shrinks all regression coefficients to zero to some degree, even those with a strong signal.
Lasso can also be used to and an order of importance for the regression coefficients by varying the strength of the L1 penalty. Regression coefficients are set to zero one-by-one as the penalty increases. Relaxed Lasso uses this rank and refits the variables without regularisation.
In Bayesian statics, regularisation is added via the prior. The Horseshoe prior can adjust to the average sparsity in the model and the Horseshoe prior either shrinks a signal aggressively to zero, or leaves the signal almost unchanged. The posterior of the model is never truly sparse. Predictive Projection can induce sparsity by setting the Monte Carlo samples of the posterior to zero for certain variables. This is done in such a way that the Kullback-Leibler divergence between the full posterior and the projected sparser posterior is minimised. I investigate the behaviour of the variable selection methods. The main focus is on the predictive performance, the sparsity, the computation time and the reliability of the estimated performance for the selected models. I apply the methods to various types of simulated data to compare the variable selection methods. The simulated data consist of data with independent predictors, collinear predictors and non-normal predictors, among others. The simulations studies show that Lasso and Predictive Projection lead to models with the highest performance overall and the predictive performance is more stable over different realisation of the data. For the same performance the Predictive Projection produces models with less variables. This makes Predictive Projection the most attractive method. I also employ the techniques to FreddieMac data, which is a data set on single-family mortgages. The results are similar to the simulated data and Predictive Projection with the Horseshoe prior is the most attractive variable selection method. Both the simulation studies and the FreddieMac application imply that the estimated performance of the Predictive Projection and Lasso are better than those of Forward Selection and Relaxed Lasso. However, the behaviour of the estimated performance remain unclear to a certain degree. More simulations per data type and more data types are needed for more insight into the estimated performance. Additional resources are needed to achieve this.