Audio-to-motion generation is an important task with applications in virtual avatar creation for XR systems and intelligent robot control in daily life scenarios.
Most current motion generation methods depend on a single encoder-decoder architecture to simultaneously model all body parts, constraining their capacity to capture the diverse and complex motions exhibited by humans.
In this paper, we propose a novel method, SpeechCAT, that employs three separate encoder-decoder modules to individually model the motions of the face, body, and hands. To capture the relationships and synchronization among these body parts, we introduce a cross-attention mechanism to effectively learn their correlations.
SpeechCAT ensures sufficient capacity to model the unique characteristics of each body part while preserving the coherence between them.
Our experimental results demonstrate the superiority of SpeechCAT over baseline methods, highlighting its effectiveness in generating diverse, realistic, and synchronized motions with face, body, and hand parts.

Early detection of leprosy, a neglected tropical disease, is crucial to preventing irreversible nerve damage and disability. Analyzing temperature vari- ations in hands using infrared (IR) cameras offers a potential low-cost alternative to existing medical equipment for early detection of leprosy. This study explores the adaptation of hand landmark detec- tion models, commonly used for hand pose track- ing, to infer the hypothenar area, a critical region for leprosy diagnosis. The research addresses the challenge of limited ground-truth data for the hy- pothenar keypoint by developing annotated datasets and evaluating machine learning models like Lasso Regression and XGBoost. These models signif- icantly outperform the existing method of linear interpolation, demonstrating the feasibility of ac- curate hypothenar keypoint prediction even with limited training data. The findings contribute to the development of accessible, automated tools for early leprosy diagnosis, particularly in resource- constrained settings.

Hand landmark detection in infrared (IR) images is essential for early leprosy diagnosis in developing countries like Nepal, helping to prevent serious complications and disability. However, current hand landmark detection models, such as Google’s detection models comprised in the MediaPipe framework, often struggle with this task due to domain mismatch. While these models are trained on RGB images, the data for this research consists of greyscale IR images. This study addresses this challenge by exploring image transformation and colorization techniques to enhance MediaPipe's hand landmark recognition accuracy on IR images. Preprocessing was chosen over retraining the existing model due to limited computational resources and the lack of labeled target domain data, which makes the retraining infeasible.

Two preprocessing pipelines were developed to address different image characteristics: images with visible hand edges but varying colors of the hand, and images where hands blend in with the background, making the edges difficult to distinguish. The transformations include turning an image into its negative, colorization, contrast enhancement using Contrast Limited Adaptive Histogram Equalization (CLAHE), and masking to remove occlusion.

To evaluate the effectiveness of these techniques, accuracy has been calculated using Percentage of Correct Keypoints (PCK) metric and were compared against two baselines: a lower bound (MediaPipe performance on unchanged IR images) and an upper bound (MediaPipe performance on similar RGB images). Preliminary findings indicate that colorization significantly improves recognition for hands with sharp color transition, while contrast enhancement boosts edge definition for hands that blend into the background. By combining these approaches, the overall accuracy of hand landmark detection improved up to 25%, depending on the threshold value, particularly for the targeted open palm-up hand position.

These results demonstrate that preprocessing techniques can effectively reduce the input domain mismatch, enhancing automated leprosy diagnosis and supporting early detection efforts in low-resource settings.

Leprosy remains a significant health challenge in developing countries, where early diagnosis is crucial to prevent severe disabilities and social stigma. Recent studies have shown that infrared imaging can be used to detect abnormalities associated with leprosy by analyzing hand temperature variations. However, existing diagnostic methods relying on manual annotation of thermal images are timeconsuming, lack standardization, and require technical expertise. This research investigates methods for implementing real-time infrared video-based temperature analysis on mobile devices by focusing on hand landmark detection models, model optimization techniques, and evaluation metrics. A comprehensive literature review identified promising models such as MediaPipe Hands, OpenPose, and YOLO variants for hand landmark detection, along with optimization methods like pruning, quantization, and Neural Architecture Search (NAS) to adapt these models for mobile deployment. Furthermore, evaluation frameworks incorporating both performance and capability-oriented metrics were examined to ensure efficient and reliable deployment on resource-constrained devices. This study provides insights into developing a fully automated, mobile-based diagnostic tool for early leprosy detection, highlighting the challenges and opportunities in adapting visual AI models for infrared analysis. Future research should focus on empirical validation of optimized models on mobile platforms.

This study investigates sensor technologies for di- agnosing leprosy in Nepal, focussing on skin tem- perature in the hands using contact and non-contact sensors. Leprosy affects the peripheral nervous system, causing thermoregulatory dysfunction de- tectable via localized skin temperature changes. A systematised comparative review compares contact thermometry, infrared (IR) thermography, and IR- RGB thermography based on measurement quality, usability, and cost. Next to the systematised re- view, an experimental method is proposed to com- bine RGB and IR imaging to enhance the spatial accuracy of automatic region of interest (ROI) de- tection using MediaPipe Hand Landmarker. The study introduces a multimodal dataset of 45 sets of annotated IR-RGB images and validates a geomet- rical image registration model, achieving 93.2% keypoint detection accuracy—significantly outper- forming IR-only sensors. Results show IR-RGB thermography as a cost-effective, flexible, and ac- curate tool for early leprosy diagnosis in resource- limited settings.

In this work, we investigate how domain adaptation techniques can improve the performance of hand landmark detection models originally trained on RGB images when deployed on infrared (IR) data. Our motivation stems from a medical use case in Nepal, where clinicians require reliable temperature estimation at hand keypoints to detect early signs of leprosy. We evaluate three methods on a small IR dataset (80 labeled images & 5000 unlabeled frames): a shallow adaptation (AdaBN), a deep alignment approach (Deep CORAL), and a test-time subspace alignment method (SSA). Our experiments show that while AdaBN and SSA yield moderate improvements, Deep CORAL achieves stronger gains through targeted training of specific model components. The combination of these methods produces superior results, yielding an 11% improvement in percentage of correct keypoints (PCK@0.05) on our custom annotated IR dataset. These findings demonstrate that combining lightweight and deep domain adaptation approaches can effectively enhance IR hand landmark detection accuracy without requiring large labeled datasets, enabling practical deployment for clinical thermal imaging in resource-limited settings.

This paper explores the challenges of converting architectural floor plans from raster to vector images. Unlike previous studies, our research focuses on domain adaptation to address stylistic and technical variations across different floor plan datasets. We develop and test our vectorization method on the CubiCasa5K benchmark, which includes 3 different floor plan styles. Our analysis reveals differences in input features across the CubiCasa5K styles, indicating the potential for domain adaptation research, mostly in room segmentation. However, we also find multiple indications that labelling in the CubiCasa5K dataset is ambiguous and inconsistent. Furthermore, styles with more training data do not always perform better, highlighting the complexity differences between floor plan styles. Our baseline shows a 0.7% gap for rooms yet a 0.6% improvement for objects, likely caused by the smaller feature gaps and inconsistent labelling. To address the adaptation gap, we add a Multi-Kernel Maximum Mean Discrepancy (MK-MMD) loss to the CubiCasa5K model to minimize feature distribution differences between domains. While our MK-MMD implementation shows potential for reducing the adaptation gap, persistence issues and mixed results across classes make it difficult to draw clear conclusions. Our findings also show the role of balancing spatial context in the MK-MMD calculation. These insights lay a foundation for future domain adaptation research in floor plan vectorization.

Object detectors have come a long way and are used for various applications. In pictures and videos, an object detector must deal with the background. In some settings, this background is indicative of the object; in others, it’s not and can even be disruptive. For models trained on data containing correlations between objects and backgrounds (background bias), it makes sense that changing the background can disrupt learned correlations. This paper is interested in how sensitive object detectors are to background changes, specifically when the training data does not contain correlations between objects and backgrounds. Models were trained on carefully controlled synthetic data, so only the backgrounds differed and correlations could be controlled. The results show that models perform better when tested with seen backgrounds than unseen backgrounds. This performance difference diminishes when the model is trained on more unique backgrounds.

It is commonly believed that image recognition based on RGB improves when using RGB-D, ie: when depth information (distance from the camera) is added. Adding depth should make models more robust to appearance variations in colors and lighting; to recognize shape and spatial relationships while allowing models to ignore irrelevant backgrounds. In this paper we investigate how robust current RGB-D models truly are to changes in appearance, depth, and background where we vary one modality (RGB or depth) and compare RGB-D to RGB-only and depth-only in a semantic segmentation setting. Experiments show that all investigated RGB-D models show some robustness to variations in color, but might severely fail for unseen variations in lighting, spatial position and backgrounds. Our results show that we need new RGB-D models that can exploit the best of both modalities while remaining robust to changes in a single modality.

Video annotation is a critical and time-consuming task in computer vision research and applications. This paper presents a novel annotation pipeline that uses pre-extracted features and dimensionality reduction to accelerate the temporal video annotation process. Our approach uses Hierarchical Stochastic Neighbor Embedding (HSNE) to create a multi-scale representation of video features, allowing annotators to efficiently explore and label large video datasets. We demonstrate significant improvements in annotation effort compared to traditional linear methods, achieving more than a 10x reduction in clicks required for annotating over 12 hours of video. Our experiments on multiple datasets show the effectiveness and robustness of our pipeline across various scenarios. Moreover, we investigate the optimal configuration of HSNE parameters for different datasets. Our work provides a promising direction for scaling up video annotation efforts in the era of video understanding.

The annotation effort associated with object detection is extremely costly. One option to reduce cost is to relax the demands on annotation quality, effectively allowing annotation noise. Current research primarily focuses on noise correction before or during training. However, there remains a gap in the research regarding the impact of specific types of human annotation noise on object-detector performance. This research aimed to determine how sensitive object detectors are to human annotation noise. A systematic methodology was developed to generate and quantify the effects of four noise types: missing annotations, extra annotations, inaccurate bounding boxes, and wrong classification labels. Additionally, evaluations were conducted on YOLOv8 and Faster R-CNN using the PASCAL VOC 2012, VisDrone, and Brain-Tumor datasets. The experiments demonstrated that adding noise to smaller datasets adversely affects the performance of object detectors trained on these datasets more than it does for those trained on larger datasets. Similarly, annotation noise in small objects affects detector performance more than large objects. Furthermore, YOLOv8 is resilient to low levels of missing annotations and inaccurate bounding boxes but is sensitive to all levels of incorrect classification labels. Interestingly, extra annotations seem to have a regularization effect on YOLOv8. In contrast, Faster R-CNN is generally more susceptible to annotation noise compared to YOLOv8, particularly concerning extra annotations, though both models display similar trends regarding inaccurate bounding boxes.

A way to reduce labelling effort and improve accuracy for object detection is class grouping. In this research, we experiment with creating hierarchical tree structures of grouped classes (super-classes). Our objective is to find out what the effects are of grouping classes in terms of accuracy and labelling effort. First we show what accuracy improvement can be gained from different grouping strategies. Then we show what the difference is in accuracy gain for a hierarchical tree structure on FasterRCNN, RetinaNet and YOLOv8. Next up we introduce a new layer in the tree structure with new super-classes and we show the difference in the tree with two and with three layers. After that we compare the accuracy for predicting only super-classes. Lastly, we want to take predicting super-classes one step further by showing that class grouping can reduce labelling effort for real life applications like autonomous cars.

Pseudo-labeling involves training models on a small amount of labeled data and then using those models' predictions on unlabeled data as labels for further training, which therefore decreases the required labeling effort. In this paper, we investigate the effects of pseudo-labeling using the robust YOLOv8 object detector. The research establishes a naive basis pseudo-labeling approach and explores improvement methods, such as object ratio based dynamic thresholds to reduce class imbalance issues, confidence scaling to retain more information when predicting a single class with less confusion, and an ensemble approach to leverage performance benefits offered by model ensembling. Experiments do not show significant improvements and mostly decreasing performance is seen on the VOC2012 dataset. However, improvements relative to the basis are observed with all proposed methods.

Object detection heavily relies on accurate annotations, which are costly to obtain but crucial for model performance. Annotation errors can severely impact the reliability of detection models. In response to this challenge, we introduce EnsembAudit (EA), a novel framework designed to autonomously identify and rectify common labeling errors, thereby reducing annotation efforts. EA leverages ensemble techniques such as Threshold Voting for error identification and Non-Maximum Suppression for error rectification.

This paper evaluates EA across various noise levels and types of labeling errors to assess its effectiveness. Our experiments demonstrate that EA excels in detecting and rectifying errors in datasets with significant noise, achieving an approximate 20% reduction in errors. However, its efficacy diminishes when applied to datasets with minimal noise. This study highlights EA's potential in enhancing annotation quality and improving the robustness of object detection applications.

Highly performing object detectors require large training datasets, which entail class and bounding box annotations. To reduce the labelling effort of curating such datasets, Weakly Supervised Object Detection is concerned with training object detectors from only class labels. The most performant weakly supervised detectors (MIL-based) have high inference times, while faster methods (CAM-based) have been primarily studied in the context of localizing just one object in an image. This research proposes an extension to weakly supervised CAM-based detectors that allows them to detect multiple objects in an image and asseses their performance at localizing the full extent of objects with bounding boxes, as well as their general location with pin-points. VGG16 and a novel FPN-based classifier are experimented with as the backbone of the network, followed by GradCAM++ which indicates through heatmaps the locations of the objects predicted by the classifiers. Additionally, the proposed method is used to create pseudo-labels on which any fully supervised detector could be trained on. Results show that while the proposed method is not suitable for detecting the full extent of objects, it can accurately pin-point their general location in near real-time, thus showing the Object is Roughly There (ORT).

The detection of changes in an area over time using remotely sensed data such as images is referred to as change detection. It has a large range of applications. For example, changes in buildings can analysed for urban planning. Many conventional image processing and machine learning-based algorithms have been developed for the purpose of change detection. Conventional non-classification algorithms have advantages in their reduced computational cost. Remotely sensed images vary in their spatial resolution, which is the area a pixel covers on the Earth surface. This work aims to explore how the spatial resolution impacts conventional non-classification pixel-based techniques in the urban change detection context, to provide insight into their performance with regards to detecting urban-related change over different resolutions. A systematic experiment is conducted by considering the LEVIR-CD and OSCD test sets in their initial as well as multiple downsampled resolutions. The change detection algorithms Change Vector Analysis (CVA) and Iteratively Reweighted Multivariate Alteration Detection (IR-MAD) are applied to the data individually. For creating binary change labels on a pixel-level, the Otsu algorithm is applied. A set of performance metrics is calculated, and trends in the metrics values over the resolutions are analysed. The data shows some trends towards improved metric values for lower spatial resolutions. The degree of the trends varies and is dependent on the algorithm and dataset. Overall, further research is necessary to consider influencing factors such as the amount of pixels in images, to refine the processing steps, and to broaden the scope of the experiment.

Change detection with remote sensing data highlights se- mantic differences in an area between two or more time intervals. It involves the comparison of aerial photographs of the same location taken some time apart. This faci itates mass scale analysis of urban and rural data over time, including population trends, city expansion trends and illegal building detection. State-of-the-art methods for the task are predominantly deep learning networks, following an encoder-decoder architecture. These architectures all share the trait of having a ”fusion” point - a location in the network where inputs transition from being processed independently to becoming correlated. F sions can be classified in three categories: early, middle and late, depending on how deep within the network they occur. This study aims to show how changing the fusion impacts the size, spread and number of changes detected. It is motivated by how the receptive field of feature maps in convolutional neural networks expands in deeper layers, extracting features with different complexities. For this, four fusion architectures on three different datasets are compared: LEVIR-CD, HiUCD and a new, fully-controled dataset, CSCD. In terms of test accuracy and the changes’ size and spread, results are inconclusive. Which fusion achieves the highest performance varies per dataset. Possible reasons why include the complexity of remote sensing data and general differences between areas, but this is a subject of further study. The only conclusive category is the number of changes detected. On aver- age, all architectures overestimate the number of changes in a scene. When the accuracy of architectures is com- parable, however, early fusion overestimates the number of objects changed the most, while middle and late fusion give more realistic estimates. The case study has room for refinement in problem isolation, more data and extending the problem towards more architectures, but is a promising step towards understanding fusion.

Urban change detection involves identifying and analyzing alterations in urban landscapes over time. This process is crucial for urban planning, environmental monitoring, and disaster management, as it provides insights into urban growth, land use changes, and human impact on the environment. This study focuses on Recurrent Neural Networks (RNNs) due to their ability to capture temporal dependencies, making them suitable for analyzing changes over time. In this research, an RNN model, specifically the SiamCRNN, was trained and evaluated on multiple datasets representing different urban scenarios. The model performed well in detecting urban changes, especially in datasets with higher spatial resolutions, but faced challenges with datasets characterized by high spectral range and complex urban structures. These findings underscore the importance of spatial resolution in influencing RNN model effectiveness for urban change detection tasks.

Masked Autoencoders (MAEs) represent a significant shift in self-supervised learning (SSL) due to their independence from augmentation techniques for generating positive (and/or negative) pairs as in contrastive frameworks. Their masking and reconstruction strategy also aligns well with SSL approaches in natural language processing. Most MAEs are built upon Transformer-based architectures where visual features are not regularized as opposed to their convolutional neural network (CNN) based counterparts, which can potentially limit their effectiveness. To address this, we introduce a novel batch-wide layer-wise regularization loss applied to representations of different Transformer layers. We demonstrate that by plugging in the proposed regularization loss, one can significantly improve the performance of MAE-based baselines.

As global urbanization reaches an all-time high, effective urban management becomes a crucial factor for efficient development. Enhanced monitoring of these transformations leads to more informed decision-making by policymakers, emphasizing the importance of tracking these changes. One method for monitoring is Change Detection (CD), which involves comparing two satellite images captured at different times to detect changes over a period of time. CD involves numerous difficulties, such as data collection, varying weather conditions, limited availability of datasets, noise, illumination differences, and discrepancies in the equipment used for image capture. Convolutional Neural Networks (CNNs) can address these issues by delivering more effective models with better performance than non-deep learning models. However, the rise of Transformers has led researchers to develop networks based on Transformer architecture, yielding more promising results than CNNs when more data is available. This paper conducts an analysis of two existing Transformer-based models, emphasizing the challenges of handling CD with artificially small datasets. Using smaller datasets reduces the requirements for remote sensing capabilities of satellites, simulating the limitations encountered during data collection and processing. The models under examination are the Bitemporal Image Transformer (BIT) and the Visual change Transformer (VcT).