Video temporal action localization is the task of identifying and localizing specific actions or activities within a video stream. Instead of only classifying which actions occur in the video stream, we aim to detect when an action begins and ends. In this work, we focus on solving this task without any supervision. Existing unsupervised methods solve this task by exploiting a combination of spatial and temporal information. We propose a new model that uses a MLP (multilayer perceptron to learn to sample prototype frames from a video. We use the distance between prototypes and video frames given by DTW (dynamic time
warping) as a loss function to update the MLP. The sampled prototypes allow us to find the start and end boundaries of actions, when combined with DTW. Additionally, the prototype frames can be used for video summarization. We analyze our model in a controlled synthetic data setup, to show the weaknesses and strengths of our models. Additionally, we use the Breakfast dataset, and Cholec80 surgery dataset to compare our model to the state-of-the-art models in a real scenario.

In this paper, we investigate the behaviour of current progress prediction methods on the currently used benchmark datasets. We show that the progress prediction methods can fail to extract useful information from visual data on these datasets. Moreover, when the methods fail to extract visual information, memory-based methods adopt a frame-counting strategy when presented with \textsl{full-video} data as input. Additionally, we evaluate all the methods on a synthetic dataset we specifically designed for the progress prediction task. On our synthetic dataset the results show that all the methods can make use of the visual information and outperform the native, non-learning baselines. We conclude that in its current form the task of progress prediction is ill-posed. The learning methods tend to fail to extract useful information from the visual data and instead rely purely on frame counting.

Regression is difficult because of noise, imbalanced data sampling, missing data, etc. We propose a method by classifying the continuous regression labels to tackle regression robustness problems. We analyze if our method can help regression, given that the class information is already included in the regression labels. We start by extensively experimenting on 1D synthetic datasets and find out that classification can help regression when the data sampling is i balanced. This happens when the data are clean, noisy in inputs and noisy in outputs, but not when they are partially missing. We then validate our conclusion on the KITTI dataset by estimating 3D object orientation. We conclude that our method can help regression in real-world.

A structured CNN filter basis allows incorporating priors about natural image statistics and thus require less training examples to learn, saving valuable annotation time. Here, we build on the Gaussian derivative CNN filter basis that learn both the orientation and scale of the filters. However, this Gaussian filter basis definition depends on a predetermined derivative order, which typically results in fixed frequency responses for the basis functions whereas the optimal frequency of the filters should depend on the data and the downstream learning task. We show that by learning the order of the basis we can accurately learn the frequency of the filters, and hence adapt to the optimal frequencies for the underlying task. We investigate the well-founded mathematical formulation of fractional derivatives to adapt the filter frequencies during training. Our formulation leads to parameter savings and data efficiency when compared to the standard CNNs and the Gaussian derivative CNN filter networks that we build on.

The Internet of Things (IoT) is a technology used in applications varying from home- and industrial automation to medical devices, smart vehicles, fitness trackers and many more. Such IoT networks often consist of incredibly resource-constrained devices, and are known as Low-power and Lossy Networks (LLNs). The Routing Protocol for LLNs (RPL) aims to provide a routing standard for such networks. Due to the tough performance constraints, RPL is unable to provide strong security guarantees. Many researchers are posing attacks against RPL-based networks, and with the increasing number of implementing devices, it is important that research is done to ensure message integrity and network reliability. In this paper we concern ourselves specifically with collusion attacks. We propose Hop-Count Reachability (HCR), a mitigating method against the coordinated blackhole attack. In HCR, leaf nodes periodically ping the root node with DAO messages. If the root node is reachable, it will reply with a DAO-ACK, upon which the leaf node sleeps for a period of time. When the number of missed ACKs in a certain time frame exceeds a certain threshold, the affected node may identify the attack and mitigate it by selecting a new parent. HCR may increase control packet overhead anywhere between 1.6 and 25\% depending on the chosen parameters, and successfully mitigates the coordinated blackhole attack in all scenarios where affected nodes can choose an (eventually) unaffected parent.

Convolutional Neural Networks are particularly vulnerable to attacks that manipulate their data, which are usually called adversarial attacks. In this paper, a method of filtering images using the Fast Fourier Transform is explored, along with its potential to be used as a defense mechanism to such attacks. The main contribution that differs from other methods that use the Fourier Transform as a filtering element in neural networks is the use of labeled data to determine how to filter the pictures. This paper concludes that, while the filtering proposed is hardly better than a simple low-pass filter, it still manages to improve resistance to adversarial attacks with a minimal drop in the standard accuracy of the network.

This research paper analyses the effect that using frequency information can have on object detectors. The latter are complex networks that learn information about objects from images and are then able to predict the location of these objects in new, unseen images. There are, however, certain datasets that are hard to learn on, partly because the environment in which images are taken is diverse and complex, and also because the objects to detect can appear in fairly different shapes. The dataset considered in this paper is called the Global Wheat Head Dataset (GWHD, provided by a Kaggle competition). An object detector is run on the original GWHD images and then the performance is compared to running the detector on a frequency filtered version of the images. A mathematical transform called Fourier Transform is used to map images from their spatial (pixel) domain to a new domain called the frequency domain, where certain non-informative frequencies are filtered out and then the images are mapped back to their spatial domain. Two experiments were conducted and results show that with this specific filtering methodology, no improvement is found on the GWHD dataset using an object detector called YoloV5. A pipeline was developed which allows for custom filtering strategy implementations and customs datasets. Similar work has shown that images in their frequency domain can speed up computational time and also increase the accuracy of an object detector, so this paper also gives the opportunity for further experiments with the created pipeline.

Wheat is among the most important grains worldwide. For the assessment of wheat fields, image detection of spikes atop the plant containing grain is used. Previous work in deep learning for precision agriculture employs the already established object detectors, Faster R-CNN and YOLO, adapted for the given context. However, these models suffer from the necessary duplicate-removal postprocessing and from the low performance on overlapping objects. On the other hand, the novel Detection Transformer (DETR) object detection approach manages to overcome such limitations, being an end-to-end anchor-free set predictor based on the transformer architecture, using the attention mechanism for modelling long-range dependencies. Consequently, the general sensitivity of this technique for small size objects in the wheat head domain is reduced. Nonetheless, previous research reflects the potential of frequency analysis techniques to increase the accuracy of a CNN. This paper aims to study the feasibility of adding frequency information as a pre-processing step to improve the performance of the DETR model for wheat head detection. Two variants of the original DETR with a mask based on the Fast Fourier Transform (FFT) of the power spectra of wheat heads and background patches are proposed and explored for improvements in prediction quality. Although promising, the best FFT-based DETR approach manages to deliver an average score of only 0.42, a slightly sub optimal performance compared to DETR’s one of 0.47. Additionally, as to grasp a sense of their capability among well-established detectors, YOLO-V3 and Faster R-CNN manage to achieve around 0.7 on the same wheat data set. Ultimately, a configurable automated overview of the development of wheat fields leads to a more efficient administration of the production process. To such end, this research explores the possible application of this new object detector in precision agriculture and provides insight into its limitations and potential ways of overcoming them.

In this paper we analyze the performance of a novel clustering objective that optimizes a neural network to predict segmentation. We challenge the reported results by replicating the original experiments and conducting additional tests to gain an insight into the algorithm. We analyzed the efficiency of the clustering objective on a different architecture, dataset and hyper-parameters. To our surprise the algorithm demonstrated considerably lower results when running on the new setup. Further, in our work we detail the reasons behind the discrepancy and provide configurations under which the method performs best. We show that the objective is highly sensitive to the type of images it is predicting and the complexity of the architecture that is being used with.

While millions of people world wide suffer from arterial diseases, such as peripheral arterial disease, there are a limited number of methods that can be used to diagnose and track these diseases which are also easy, quick and non-invasive.

This work focuses on what is needed to improve diagnosing and tracking of peripheral arterial disease (PAD) using visualisation techniques. Visualising the blood flow pulsation in the skin can be useful in cases of arterial diseases, as the diseases can influence the blood flow by obstructions and arterial stiffness. The main objective in the visualisation is to show the acceleration of the blood flow, as this is linked to the arterial stiffness.

The proposed algorithm for visualising the acceleration of the blood flow is comprised of multiple steps, including techniques such as motion reduction, Eulerian video magnification, remote photoplethysmography signal extraction and using the second derivative. The input of this algorithm are videos of the skin of patients, this makes this method easy and non-invasive.

Photoplethysmography (PPG) signals are present in videos of skin, but can not be seen with the naked eye. Using Eulerian video magnification the PPG signals are amplified for better processing and visibility. By combining groups of pixels into small patches, a decrease in processing time is achieved and it adds a filtering effect. The size of the patches controls the resolution of the visualisation. Movement from the camera or patient is detrimental when extracting the PPG signal from the video. To counteract the motion in the videos a motion reduction step, using optical flow, is applied. Using the Plane-orthogonal-to-skin (POS) algorithm, the signal extracted from videos is converted to a PPG signal. Calculating the second derivative of the PPG signal gives the acceleration of the signal. By splitting the acceleration signal into positive and negative numbers, the acceleration and deceleration of the blood flow is visualised.

Synthetic videos simulating the skin were generated in various levels of accuracy to aid the development of the algorithm and to conduct experiments. The levels range from a simple pulsating square to a moving PPG signal over a blood vessel like structure. In addition, real videos of patients were used.

The experiments show the feasibility of visualising the acceleration of the blood flow pulsation in the skin, but also highlights areas of improvements and future research. More fine-tuning of the algorithm is needed, in addition to acquiring more videos of patients with PAD before and after surgery in a controlled environment.

A working proof of concept of the algorithm is shown. It has the potential of being a novel method of diagnosing and tracking arterial diseases.

Annotating a large-scale in-the-wild person re-identification dataset especially of marathon runners is a challenging task. The variations in the scenarios such as camera viewpoints, resolution, occlusion, and illumination make the problem non-trivial. Manually annotating bounding boxes in such large-scale datasets is cost-inefficient. Additionally, due to crowdedness and occlusion in the videos, aligning the identity of runners across multiple disjoint cameras is a challenge. We collected a novel large-scale in-the-wild video dataset of marathon runners. The dataset consists of hours of recording of thousands of runners captured using 42 hand-held smartphone cameras and covering real-world scenarios. Due to the presence of crowdedness and occlusion in the videos, the annotation of runners becomes a challenging task. We propose a new scheme for tackling the challenges in the annotation of such large dataset. Our technique reduces the overall cost of annotation in terms of time as well as budget. We demonstrate performing fps analysis to reduce the effort and time of annotation. We investigate several annotation methods for efficiently generating tight bounding boxes. Our results prove that interpolating bounding boxes between keyframes is the most efficient method of bounding box generation amongst several other methods and is 3x times faster than the naive baseline method. We introduce a novel way of aligning the identity of runners in disjoint cameras. Our inter-camera alignment tool integrated with the state-of-the-art person re-id system proves to be sufficient and effective in the alignment of the runners across multiple cameras with non-overlapping views. Our proposed framework of annotation reduces the annotation cost of the dataset by a factor of 16x, also effectively aligning 93.64\% of the runners in the cross-camera setting.

We focus on the problem of estimating human hand-tremor frequency from input RGB video data. Estimating tremors from video is important for non-invasive monitoring, analyzing and diagnosing patients suffering from motor-disorders such as Parkinson’s disease. We consider two approaches for hand-tremor frequency estimation: (a) a Lagrangian approach where we detect the hand at every frame in the video, and estimate the tremor frequency along the trajectory; and (b) an Eulerian approach where we first localize the hand, we subsequently remove the large motion along the movement trajectory of the hand, and we use the video information over time encoded as intensity values or phase information to estimate the tremor frequency. We estimate hand tremors on a new human tremor dataset, TIM-Tremor, containing 55 tremor patient recordings together with: associated ground truth accelerometer data from the most affected hand, RGB video data, and aligned depth data, as well as a multitude of more dynamic tasks, involving larger motion of the hands.

Action recognition continues to receive significant attention from the research community, with new neural network architectures being developed continuously. Optical flow is by far the most popular input motion representation to these architectures, leaving a lot of undiscovered potential for other types of motion representations. Eulerian representations have recently showed huge improvements in areas like motion magnification and video frame interpolation. This work proposes using a phase-based approach to make the best out of Eulerian motion information. We do this by learning complex filters using complex convolutional layers. Phase descriptions are extracted from the feature maps of these complex layers, and are then passed to the remainder of the convolutional network. Our approach shows great potential, and its performance exceeds that of a single optical flow frame input. We provide detailed analysis on using phase-based methods for Eulerian representations, in addition to further analysis on using Eulerian phase, rather than Lagrangian optical flow, for action recognition.

Over the past several years, deep and wide neural networks have achieved great success in many tasks. However, in real life applications, because the gains usually come at a cost in terms of the system resources (e.g., memory, computation and power consumption), it is impractical to run top-performing but heavy networks such as VGGNet and GoogleNet directly on mobile and embedded devices, like smartphones and cameras. To tackle this problem, we propose the use of recurrent layers in residual networks to reduce the redundant information and save the parameters. Furthermore, with the help of feature map knowledge transfer, the performance of Recurrent Residual Networks (ReResNet) can be improved so as to reach similar accuracy to some complex state-of-the-art architectures on CIFAR-10, even with much fewer parameters. In this paper, we demonstrate the efficiency of ReResNet possibly improved by Feature Transfer on three datasets, CIFAR-10, Scenes and MiniPlaces.

In this thesis, we propose a novel unsupervised clean-noisy datasets adaptation algorithm based on standard deep learning networks. Specifically, we jointly learn a shared feature encoder for two tasks: 1)supervised classification trained on labeled source (clean) dataset, and 2) unsupervised adaptation to map discriminant features from both source and target domains to a common space. Our proposed network is optimized by a step backpropagation similarly as some of the Generative Adversarial Networks (GANs).

We evaluate our proposed network on two datasets, where a improvement of classification performance is achieved (up to ~19% in average accuracy over all noise levels) over state-of-the-art denoising algorithm BM3D. Interestingly, we also observe that our proposed approach improves the feature transferability on deep networks with its task-specific learning steps.