3D object detection models that exploit both LiDAR and camera sensor features are top performers in large-scale autonomous driving benchmarks. A transformer is a popular network architecture used for this task, in which so-called object queries act as candidate objects. Initializing these object queries based on current sensor inputs leads to state-of-the-art performance. Existing methods rely strongly on LiDAR data however, and do not fully exploit image features. Besides, they introduce significant latency.

To overcome these limitations we propose EfficientQ3M, an efficient, modular, and multimodal solution for object query initialization for transformer-based 3D object detection models. Using both the LiDAR and camera modalities as input, we use efficient grid sampling and a lightweight detection head to predict a set of initial object query locations and corresponding query feature vectors. The proposed initialization method is combined with a “modality-balanced” transformer decoder where the queries can access all sensor modalities throughout the decoder.

We achieve state-of-the-art performance for both LiDAR-camera and LiDAR-only sensor setups on the competitive nuScenes benchmark while being up to 15 times more efficient than the closest related method. The proposed initialization can be applied with any combination of sensor modalities as input, demonstrating its modularity.

This research investigates and describes an image search engine for digital history using deep learning technologies. It is part of the Engineering Historical Memory research, contributing to a multilingual and transcultural approach to decode-encode the treasure of human experience and transmit it to the next generation of world citizens. The engine provides a new way to search in online (historical) digital libraries using content-based image retrieval and makes linguistic metadata redundant. State-of-the-art deep learning methodologies in computer vision have been investigated and tested. These methodologies include both template-based matching and feature-based matching. A VGG16 Convolutional Neural Network based approach, called D2-Net, is concluded to provide the best basis. D2-Net is then further analyzed, improved, and optimized to run on a large dataset of more than 12k image combinations related to history, heritage, and art. The final implementation shows promising results with a precision of 0.96 and a recall of 0.44 on a challenging testing dataset. Future improvements include speed improvement and model training.