Deformable Image Registration (DIR) is a medical imaging process involving the spatial alignment of two or more images using a transformation model that can account for non-rigid deformations. B-spline-based transformation models have emerged as a common approach to express such spatial alignments. However, without additional measures, their flexibility can lead to physically implausible deformations. This flexibility has motivated the inclusion of penalty terms to improve the smoothness and regularity of the transformation. Determining an appropriate weight for this penalty term is difficult, as each registration problem requires a different trade-off between this penalty term and the quality of the transformed image.

Gradient-based methods are commonly used as optimization methods in medical image registration toolboxes due to their computational efficiency and fast convergence rates. However, due to their gradient-based approach, they may converge prematurely in local minima. In this thesis, we investigate the efficacy of a gradient-less alternative: the Real-Valued Gene-pool Optimal Mixing Evolutionary Algorithm (RV-GOMEA), a population-based method that can exploit the problem structure of optimization problems through explicit mappings of dependencies between problem variables. To improve the computational efficiency of RV-GOMEA when applied to DIR, we show how to apply partial evaluations for common image similarity metrics and penalty terms when using B-spline-based transformation models.

We test RV-GOMEA on a synthetic registration problem to better understand its performance in the context of DIR. Based on our findings, we propose several methods that hybridize RV-GOMEA with a gradient-based method and impose specific constraints on the B-spline-based transformation model. We validate the performance of these methods on clinical registration problems and find that RV-GOMEA with a gradient-based local search operator can provide significant benefits over purely gradient-based methods for DIR problems. Additionally, placing specific constraints on the transformation model can increase the regularity of transformations without requiring a penalty term.

Even if a Multi-modal Multi-Objective Evolutionary Algorithm (MMOEA) is designed to find all locally optimal approximation sets of a Multi-modal Multi-objective Optimization Problem (MMOP), there is a risk that the found approximation sets are not smoothly navigable because the solutions belong to various niches, which reduces the insight for decision makers. Moreover, when the multi-modality of MMOPs increases, this risk grows and the trackability of finding all locally optimal approximation sets decreases. One example where this issue occurs is that of High-Dose-Rate (HDR) brachytherapy for prostate cancer. In HDR brachytherapy a treatment plan is to be optimized that irradiates a tumour with a prescribed dose, whilst sparing all of the healthy organs surrounding the tumour. The radiation is administered through a radioactive source that is stopped at certain dwell positions within a set of hollow catheters that have been implanted into the patient. In a treatment plan, each of the dwell positions is given a specific dwell time for which the source is kept at that location in order to irradiate the surrounding tissue.
To tackle the navigability issues, two new MMOEAs are proposed: Multi-Modal Bézier Evolutionary Algorithm (MM-BezEA) and Set Bézier Evolutionary Algorithm (Set-BezEA). Both MMOEAs produce approximation sets that cover individual niches and exhibit inherent decision-space smoothness as they are parameterized by Bézier curves. MM-BezEA combines the concepts behind the recently introduced BezEA and MO-HillVallEA to find all locally optimal approximation sets. Set-BezEA employs a novel multi-objective fitness function formulation to find limited numbers of diverse, locally optimal, approximation sets for MMOPs of high multi-modality.
Both algorithms, but especially MM-BezEA, are found to outperform the MMOEAs MO_Ring_PSO_SCD and MO-HillVallEA on MMOPs of moderate multi-modality with linear Pareto sets. Moreover, for MMOPs of high multi-modality, Set-BezEA is found to indeed be able to produce high-quality approximation sets, each pertaining to a single niche. Set-BezEA is also shown to be comparable to the current BRIGHT approach used in the Amsterdam UMC for the optimization of treatment plans for prostate cancer HDR brachytherapy, which opens the way for it to be introduced in the clinical practice in the future.