Radar-based Human Activity Recognition(HAR) is considered by using snapshots of point clouds. Such point cloudsinterpret 2D images generated by an mm-wave FMCW MIMO radar enriched byincluding Doppler and temporal information. We use the similarity between suchradar data representation and the core of the self-attention concept inartificial intelligence. Three self-attention models (Point Transformer) areinvestigated to classify Activities of Daily Living (ADL). An experimentaldataset collected at TU Delft is used to explore the best combination ofdifferent input features, the effect of a proposed Adaptive ClutterCancellation (ACC) method, and the robustness in a leave-one-subject-outscenario. Results with a macro F1 score in the order of 90% are demonstratedwith the proposed method, including activities that are static postures withlittle associated Doppler.@en

Driven by its contactless sensing capabilities and the lack of optical images being recorded, radar technology has been recently investigated in the context of human healthcare. This includes a broad range of applications, such as human activity classification, fall detection, gait and mobility analysis, and monitoring of vital signs such as respiration and heartbeat. In this paper, a review of notable achievements in these areas and open research challenges is provided, showing the potential of radar sensing for human healthcare and assisted living.

@en

This thesis introduces the relevance of radar systems in the realm of human activity recognition (HAR) in Chapter 1. The study touches upon the complex understanding of continuous human activities and the existing challenges and gaps in current methodologies, hinting at the innovative technical approaches that are to be detailed in the following chapters. The technical foundation of the research is given in Chapter 2 by introducing distributed ultrawideband (UWB) radar systems. These systems, especially when spatially distributed, bring a depth of information by integrating data from multiple radar nodes and spatial perspectives. There is a significant emphasis on how different fusion techniques, both late and early, play a crucial role in harnessing data effectively, particularly in the context of HAR. A critical contribution in the study is the potential to deviate from conventional radar data domains, such as microDoppler spectrograms for activity recognition. The research in Chapter 3 highlights an alternative approach, rooted in the radar phase information from a highresolution rangetime map, which bypasses the limitations of common FFTbased radar data domains. This methodology, paired with the histogram of oriented gradients (HOG) algorithm, showcases promising results that can be particularly interesting for realtime applications with computational constraints. The research in Chapter 4 underlines the efficacy of employing a network of spatially distributed UWB radars for continuous HAR. These networks address the downsides of using a single sensor, like unfavorable aspectangle observations. The study delves into fusion methodologies and their implementation in classifying activities, particularly using recurrent neural networks. To assess these continuous recognition systems, novel evaluation metrics are proposed, offering a deeper insight into the practicality and effectiveness of such systems with temporal classification capabilities. Indoor radar networks often face multipath challenges. The study in Chapter 5 not only identifies this challenge, but also uses the multipath components by leveraging these typically unwanted phenomena to enhance classification capabilities. Through a pipeline that isolates, determines, and analyzes different propagation pathways, there is an evident boost in the network’s perception. This novel approach showcases a significant performance upward trend, especially when employing convolutional neural networks. Chapter 6 of the research focuses on the complexities of HAR in crowded environments. The study introduces the challenges of differentiating the activities of walking versus standing idle for multiple individuals simultaneously. The investigation shows initial promising results by using synthetic data generated from experimental recordings, by employing a regressionbased approach and leveraging diverse techniques such as LSTM, CNN, SVM, and linear regression. In conclusion, the research offers a reflective glance at the breakthroughs achieved in the domain of radarbased HAR in Chapter 7. The significant contributions and advancements of the study are highlighted. Looking ahead, the chapter identifies research areas for exploration and further improvement.@en

Radar-based human activity recognition in crowded environments using regression approaches is addressed. Whereas previous research has focused on single activities and subjects, the problem of continuous activity recognition involving up to five individuals moving in arbitrary directions in an indoor area is introduced. To treat the problem, a regression-based approach is used, which offers innovative insights into creating robust and accurate systems for monitoring human activities.Novel approaches utilizing LSTM or CNN regression techniques with Linear Regression and Support Vector Machine regressor are compared on extracted features from radar data through the Histogram of Oriented Gradients and Principal Component Analysis. These approaches are rigorously evaluated by a Leave-One-Group-Out method, with performance assessed using common regression metrics such as the RMSE. The most promising outcomes were observed for crowds of three and five individuals, with respective RMSE of approximately 0.4 and 0.6. These results were primarily achieved by utilizing the micro-Doppler (µD) Spectrogram or range-Doppler data domain.@en

Radar-based human motion and activity recognition is currently a topic of great research interest, as the aging population increases and older individuals prefer an independent lifestyle. This technology has a wide range of applications, such as fall detection in assisted living, gesture recognition for human-machine interfaces, and many more. Numerous studies exist on various approaches for radar-based activity capture and classification. However, most of these employ rather artificial data, often obtained in laboratory environments, and typically collected under particular conditions. Specifically, most research so far has aimed at distinguishing a predefined set of single activities with a defined start, stop and duration. This paper aims at drawing the attention to a so far less researched issue, one that will be of vital importance for future real-world application of radar-based human activity recognition: continuous activity recognition, i.e. recognizing specific activities in a stream of several sequential activities with unknown duration and arbitrary transitions between different classes of activities. A review on the current state of the art in this relatively new topic is given, followed by a discussion on future research directions.@en

Recognition of continuous human activities is investigated in unconstrained movement directions using multiple spatially distributed radar nodes, where activities can occur at unfavourable aspect angles or occluded perspectives when using a single node. Furthermore, such networks are favourable not only for the aforementioned aim, but also for larger controlled surveillance areas that may require more than just one sensor. Specifically, a distributed network can show significant differences in signature between the nodes when targets are located at long distances and different aspect angles. Radar data can be represented in various domains, where a widely known domain for Human Activity Recognition (HAR) is the microDoppler spectrogram. However, other domains might be more suitable for better classification performance or are superior for low-cost hardware with limited computational resources, such as the Range-Time or the Range-Doppler domain. An open question is how to take advantage of the diversity of information extractable from the aforesaid data domains, as well as from different distributed radar nodes that simultaneously observe a surveillance area. For this, data fusion techniques can be used at both the level of data representations for each radar node, and across the different nodes in the network. The introduced methods of decision fusion, where typically one classifier operates on each node, or feature fusion, where the data is concatenated before using one single classifier, will be exploited, investigating their performance for continuous sequence classification, a more naturalistic and realistic way of classifying human movements, also accounting for inherent imbalances in the dataset. @en

Continuous Human Activity Recognition (HAR) in arbitrary directions is investigated using 5 spatially distributed pulsed Ultra-Wideband (UWB) radars. Such activities performed in arbitrary and unconstrained trajectories render a more natural occurrence of Activities of Daily Living (ADL) to be recognized. An innovative signal level fusion method was applied on the Range-Time (RT) maps, and deep learning classification via Recurrent Neural Networks (RNN) with and without bidi-rectionality was used on the computed micro-Doppler (μD) spectrogram. To assess classification performances, novel evaluation metrics accounting for the continuous nature of the sequence of activities and for imbalances in the dataset are proposed and compared with existing metrics. It is shown that conventional accuracy evaluation is too coarse, and that the proposed metrics need to be considered for a more comprehensive evaluation.@en

The feasibility of classifying human activities measured by a distributed ultra-wideband (UWB) radar system using Range-Doppler (RD) images as the input to classifiers is investigated. Kinematic characteristics of different human activities are expected to be captured in high-resolution range-Doppler images measured by UWB radars. To construct the dataset, 5 distributed monostatic Humatics P410 radars are used to record 15 participants performing 9 activities in arbitrary directions along a designated trajectory. For the first time a convolution-free neural network based on the novel multi-head attention mechanism (the Vision Transformer architecture) is adopted as the classifier, attaining an accuracy of 76.5 %. A comparison between Vision Transformer and more conventional CNN-based architectures, such as ResNet and AlexNet, is also conducted. The robustness of Vision Transformer and the other networks against unseen participants is also validated by testing via Leave One Participant Out validation.@en

Continuous Human Activity Recognition (HAR) in arbitrary directions is investigated in this paper using a network of five spatially distributed pulsed Ultra-Wideband radars. While activities performed continuously and in unconstrained trajectories provide a more realistic and natural scenario for HAR, the network of radar sensors is proposed to address the issue of unfavourable or occluded perspectives when using only a single sensor. Different techniques to combine the relevant information from the multiple radars in the network are investigated, focussing on signal level fusion directly applied on Range-Time maps, and the selection of radar nodes based on location and velocity of the target derived from multilateration processing and tracking. Recurrent Neural Networks with and without bidirectionality are used to classify the activities based on the micro-Doppler (μD) spectrograms obtained for sensor fusion techniques. To assess classification performances, novel evaluation metrics accounting for the continuous nature of the sequence of activities and inherent imbalances in the dataset are proposed and compared with existing metrics. It is shown that the conventional accuracy metric may not capture all the important aspects for continuous HAR, and the proposed metrics can be considered for a more comprehensive evaluation.

@en

Human activities classification in assisted living is one of the emerging applications of radar. The conventional analysis considers micro-Doppler signatures as the chosen input for feature extraction or deep learning classification algorithms, or, less frequently, other radar data formats such as the range-time, the range-Doppler, or the Cadence Velocity Diagram. However, these data are typically used as real-valued images, whereas they are actually complex-valued data structures. In this paper, neural networks processing radar data as complex data structures are investigated, with a focus on spectrograms, range-time, and range-Doppler plots as the data formats of choice. Different network architectures are explored both in terms of complex numbers' representations and the depth/complexity of the architecture itself. Experimental data with 9 activities and 15 volunteers collected using an UWB radar are used to test the networks' performances. It is shown that for certain data formats and network architectures, there is an advantage in using complex-valued networks compared to their real-valued counterparts.@en

Unconstrained human activities recognition with a radar network is considered. A hybrid classifier combining both convolutional neural networks (CNNs) and recurrent neural networks (RNNs) for spatial–temporal pattern extraction is proposed. The 2-D CNNs (2D-CNNs) are first applied to the radar data to perform spatial feature extraction on the input spectrograms. Subsequently, gated recurrent units with bidirectional implementations are used to capture the long- and short-term temporal dependencies in the feature maps generated by the 2D-CNNs. Three NN-based data fusion methods were explored and compared with utilize the rich information provided by the different radar nodes. The performance of the proposed classifier was validated rigorously using the K-fold cross-validation (CV) and leave-one-person-out (L1PO) methods. Unlike competitive research, the dataset with continuous human activities with seamless interactivity transitions that can occur at any time and unconstrained moving trajectories of the participants has been collected and used for evaluation purposes. Classification accuracy of about 90.8% is achieved for nine-class human activity recognition (HAR) by the proposed classifier with the halfway fusion method@en

Continuous Activities of Daily Living (ADL) recognition in an arbitrary movement direction using five distributed pulsed Ultra-Wideband (UWB) radars in a coordinated network is proposed. Classification approaches in unconstrained activity trajectories that render a more natural occurrence for Human Activity Recognition (HAR) are investigated. Feature and decision fusion methods are applied to the priorly extracted handcrafted features from the range-Doppler. A following multi-nomial logistic regression classifier, commonly known as Softmax, provides explicit probabilities associated with each target label. The outputs of these classifiers from different radar nodes were combined with a probability prediction balancing approach over time to improve performances. The final results show average improvements between 6.8% and 17.5% compared to the usage of any single radar in unconstrained directions@en

We consider Activities of Daily Living (ADL) as actions that are continuously monitored and cannot easily be separated in time. Both, the micro-Doppler signature and range-map are used to determine transitions from translation (walking) to in-place activities and vice versa, as well as to provide activity onset and offset times. The in-place action classes after and before walking are handled by forward and backward in time classifiers. The paper extends the recently published work on ADL based human states and transitioning activities, where time separable and inseparable activities were exploited. The previous work has shown that limiting the classes of activities, which are physically possible and associated with the current state, improves the classification rates compared to considering all possible ADL in the dataset at any given time. These approaches are suitable for switching classification methods to achieve superior classification results.

@en

We perform classification of Activities of Daily Living (ADL) using a Frequency-Modulated Continuous Waveform (FMCW) radar. In particular, we consider contiguous motions that are inseparable in time. Both the micro-Doppler signature and range-map are used to determine transitions from translation (walking) to in-place motions and vice versa, as well as to provide motion onset and offset times. The in-place motion classes post and prior to walking can be separately handled by forward and backward in time classifiers. The paper describes ADL in terms of motion states and transitioning motions or activities, and sets forward a framework to deal with time separable and inseparable activities. It is shown that limiting the classes of activities, which are physically possible and associated with the current state, improves the classification rates compared to incorporating all ADL for any given time.@en

The authors consider radar classifications of activities of daily living, which can prove beneficial in fall detection, analysis of daily routines, and discerning physical and cognitive human conditions. They focus on contiguous motion classifications, which follow and commensurate with the human ethogram of possible motion sequences. Contiguous motions can be closely connected with no clear time gap separations. In the proposed approach, they utilise the Radon transform applied to the radar range‐map to detect the translation motion, whereas an energy detector is used to provide the onset and offset times of in‐place motions, such as sitting down and standing up. It is shown that motion classifications give different results when performed forward and backward in time. The number of classes, thereby classification rates, considered by a classifier, is made varying depending on the current motion state and the possible transitioning activities in and out of the state. Two different examples are given to delineate the performance of the proposed approach under typical sequences of human motions.@en

Micro-Doppler spectrograms are a conventional data representation domain for movement recognition such as Human Activity Recognition (HAR) or gesture detection. However, they present the problem of time-frequency resolution trade-offs of Short-Time Fourier Transform (STFT), which may have limitations due to unambiguous Doppler frequency, and the STFT computation may be onerous in constrained embedded environments. We propose in this paper an alternative classification approach based on the radar phase information directly extracted from high-resolution Range Map (RM). This novel approach does not use the aforementioned micro-Doppler processing, and yet achieves equivalent or even superior classification results. This shows a potential advantage for low-latency, real-time applications, or computationally constrained scenarios. The proposed method exploits the Histogram of Oriented Gradients (HOG) algorithm as an effective feature extraction algorithm, specifically its capability to capture the unique shape and patterns present in the wrapped phase domains, such as their contour intensity and distributions. Validation results consistently above 92% demonstrate the effectiveness of this method on two independent datasets of arm gestures and gross-motor activities. These were classified with three algorithms, namely the Nearest Neighbor (NN), the linear Support Vector Machine (SVM), and the Gaussian SVM classifiers using the proposed phase information. Feature fusion of different data domains, e.g. the modulus of the RM fused with the RM phase information, is also investigated and shows classification improvement specifically for the robustness of activity performances, such as the aspect angle and the speed of performance.@en

In this paper, we investigate the classification of Activities of Daily Living (ADL) by using a pulsed ultra-wideband radar. Specifically, we focus on contiguous activities that can be inseparable in time and share a common transition, such as walking and falling. The range-time data domain is deliberately exploited to determine transitions from translation activities to in-place activities and vice versa, using a simple, yet effective approach based on the proposed Derivative Target Line (DTL). The separation of different in-place activities is then addressed using an energy detector finding the onset and offset times. Furthermore, the possible ADL for classification are limited at any decision stage based on kinematic constraints of human movements. We show that such limitation of classes at any given time leads to a classification improvement over a classifier containing always all ADL classes.@en