Critically Pre-Trained Neural Cellular Automata as Robot Controllers

Abstract

Neural Cellular Automata (NCA) have recently been proposed as neuromorphic robot controllers. Despite their promising behavioural characteristics and small parameter counts, training NCA for control tasks has proven difficult and unstable. It is so tricky that curriculum-like multi-stage training programs must be used for simple control tasks. This thesis posits how criticality theory, an amalgam of statistical mechanics and neuroscience, presents a compelling case for pre-training NCA into a critical state. Mainly the increase in inter-cellular com-munication distance and maximization of available information. This thesis presents a novel NCA update function architecture loosely based on neuroscience and two novel interchangeable pre-training methods, one implicit and one explicit, based on criticality theory aimed at improving training performance. The new architecture and pre-training methods are tested on the Cart-Pole environment and trained with Double Deep Q-Learning (DDQN) and neuro-evolution. The new architecture improves the training speed and general performance of the NCA, whilst the two pre-training steps greatly stabilise the control task training when using DDQN. The explicit methods resulted in faster agent training than the implicit methods, but the pre-training step was prone to failure, whereas the implicit methods always succeeded. Neuro-evolution was more efficient than DDQN when training iterations and helped explain the dynamics that make NCA challenging to train. The architecture and pre-training steps were further tested on the LunarLander problem, a more complex control task. The neuro-evolution method succeeded in training but did not present excellent results, whilst DDQN failed outright to train.