Deep Neural Networks (DNNs) have the potential to make various clinical procedures more time-efficient by automating medical image segmentation; largely due to their strong, in some cases human-level, performance. The design of the best possible medical image segmentation DNN, however, is task-specific. Neural Architecture Search (NAS), i.e., the automation of neural network design, has been shown to have the capability to outperform manually designed networks. However, the existing NAS methods for medical image segmentation have explored a quite limited range of types of DNN architectures that can be discovered, and, more importantly, do not evaluate the accuracy of performance estimation methods.

In this thesis, various performance estimation methods for DNNs are analysed for medical image segmentation tasks. Due to the use of different metrics, small datasets, and inter-physician variability, DNN performance values are susceptible to considerable noise. Through experiments on multiple datasets, it is shown that performance estimation needs to be more elaborate than proposed in previous literature on NAS for medical image segmentation. Only then can the noise induced by the problem be overcome. Based on evaluations of NAS performance with different levels of noise, a method is put forward to evaluate this noise, such that a more informed decision on performance estimation can be made.

The second contribution of this thesis, is the proposal of a novel NAS search space for medical image segmentation networks. This search space combines the strength of a generalised encoder-decoder structure, well known from U-Net, with network blocks that have been proven to have a strong performance in image classification tasks. The search is performed by looking for the best topology of the network, and simultaneously searching the configuration of each cell. This allows for interactions between topology- and cell-level attributes to be found. Experiments were performed on two publicly available datasets. The networks discovered by the proposed NAS method perform better than well-known handcrafted segmentation networks, and outperform networks found with other NAS approaches that perform only topology search, and topology-level search followed by cell-level search.

Finally, three search algorithms are compared for different performance estimation methods on a realistic clinical medical image segmentation task. The results show that the performance of these algorithms is very similar in noisy environments for initial runs, and show deterioration of performance for all algorithms when correlation with the validation performance values are low. This supports the findings that not adapting performance estimation to the task at hand will lead to poor NAS performance, no matter the chosen search algorithm.

Computer vision tasks, like supervised image classification, are effectively tackled by convolutional neural networks, provided that the architecture, which defines the structure of the network, is set correctly. Neural Architecture Search (NAS) is a relatively young and increasingly popular field that is concerned with automatically optimizing the architecture of neural networks. Previously known work shows that even though a recent trend has been to develop increasingly complex search strategies for NAS, several search strategies do not significantly outperform simple approaches like randomly sampling from the search space on single-objective NAS tasks. Additionally, proper ablation studies are often missing. Therefore, it is currently uncertain at best which mechanisms are key for an algorithm to have to achieve excellent NAS performance. In the first part of this thesis, Local Search (LS) and a differently biased form of random search, are proposed for multi-objective (MO) NAS. The multi-objective version of NAS is studied less and understanding the trade-off between between multiple objectives for architectures is arguably more interesting. We find that very simple algorithms can achieve search performance close to that of state-of-the-art evolutionary algorithms (EAs), while outperforming plain random search. Additionally, we find that the quality of the set of architectures found by LS is similar to the those found by the EAs, if compared with respect to test accuracy. Nevertheless, from the compared search strategies the Multi-Objective Gene-pool Optimal Mixing Evolutionary Algorithm (MO-GOMEA), a state-of-the-art model-based EA, achieves the best performance. In the second part of this thesis, it is explored which mechanisms are essential for MO-GOMEA to achieve an excellent search performance for NAS spaces. We find that the automatic population-sizing scheme of MO-GOMEA offers a welcome anytime-performance, but objective space clustering has only a small beneficial impact. The number of clusters can be set arbitrarily. Special (extreme) clusters that optimize for one objective only can be enabled to the practitioner’s preference, resulting in different search behaviors. The improvement in performance gained by automatically detecting and exploiting dependencies within architectures is limited: this model-based aspect of MO-GOMEA seems only helpful for finding highly accurate networks.