It is commonly believed that image recognition based on RGB improves when using RGB-D, ie: when depth information (distance from the camera) is added. Adding depth should make models more robust to appearance variations in colors and lighting; to recognize shape and spatial relationships while allowing models to ignore irrelevant backgrounds. In this paper we investigate how robust current RGB-D models truly are to changes in appearance, depth, and background where we vary one modality (RGB or depth) and compare RGB-D to RGB-only and depth-only in a semantic segmentation setting. Experiments show that all investigated RGB-D models show some robustness to variations in color, but might severely fail for unseen variations in lighting, spatial position and backgrounds. Our results show that we need new RGB-D models that can exploit the best of both modalities while remaining robust to changes in a single modality.

Implicit neural representations (INRs) exhibit exceptional compression and generalisation abilities that have enabled striking progress across a variety of applications. These properties have fuelled a growing interest in leveraging INRs for traditional classification tasks as a memory-efficient alternative representation of images, breaking the persistent link between image resolution and associated resource costs. Current INR classification methods face limitations such as a restriction to low-resolution data and sensitivity to image-space transformations. We attribute these issues to the employed INR architecture which lacks mechanisms for local representation, thereby disregarding spatial structure within the data and furthermore limiting their ability to capture high-frequency details. In this work, we propose ARC: Anchored Representation Clouds, a novel INR architecture that explicitly anchors latent vectors in image-space. By introducing spatial structure to the latent vectors, ARC can capture local image data which in our testing leads to state-of-the-art implicit image classification of both low- and high-resolution images and increased robustness against image-space translation.