A widely held assumption within the field of Trajectory Prediction is the perfect and complete observation of agents’ pasts. While this assumption allows for a simpler representation of the prediction problem, it no longer holds true when prediction models are expected to operate on histories generated by upstream perception systems, which are susceptible to fail. Occlusions are a particularly important cause of perception failures. They fragment tracked agents’ trajectories, and can often hide their most recent position(s) from the perception system. While most prediction models that are currently being researched cannot account for the possible incompleteness of agents’ past histories, we devise a prediction model that is designed to directly operate on partially missing histories caused by occlusions. We present OcclusionFormer, a TransFormer based prediction model, which predicts agent’s futures from their last observed position, without requiring imputation of missing past positions. Experiments show that our design is occlusion capable, as it can predict from trajectories with partially missing data, while remaining performant in the ideal, fully observed scenario. We also conduct research on the integration of an occlusion map within our model, which could help narrow the region of plausible prediction for occluded agents. We observe that, while the addition of such a map does improve the coherence of predictions with respect to the configuration of the occluded space, it results in a degradation of prediction performance.

The world is heading more and more towards automation, that goes for transportation as well. Various car manufactures already have released level 2 autonomous vehicles meaning that the future is not that far away. An essential part of driving is of course detecting and obeying the traffic signs. Advanced Driver Assistance Systems (ADAS) like speed limit detection systems or a more advanced variant the Traffic Sign Detection (TSD) systems are integrated in top-of-the-line car models in order to increase safety and relieve some ofthe drivers workload. These systems attempt to make drivers aware of incoming signs on the road and warn them against possible danger or in some cases even take control over the vehicle. As in many computer vision systems differences in lighting, weather conditions, motion blur, and poor image quality can reduce the effectiveness of these systems. In case of the Traffic Sign Detection systems other challenges arise like differences in sign position and in the condition (dirty, deformed, discolored etc.) of the sign itself. In order to fully rely on such system all these challenges have to be overcome. The goal of this thesis is to enhance a traffic sign detection model that requires relatively low processing power in such a manner that it can compete with traffic sign models that show great results and achieve high accuracy but require high processing power maintaining the ability to operate in real time. The suggested BoBT method increases the accuracy of a model at the cost of inference speed. There is a small difference in precisely localizing the traffic signs compared to state of the art models but in terms of detecting traffic signs the difference is less than a percent with the highest accuracy model. Not to mention that it still outperforms these models in terms of speed.

For an Autonomous Vehicle (AV) to traverse safely in traffic, It is vital it can anticipate the behavior of surrounding traffic participants using motion prediction. Current motion prediction approaches can be categorized into object-centered and object-agnostic methods and are primarily based on deep learning. The former relies on a human-engineered pipeline of object detection and tracking, of which the errors can accumulate in the motion predictions. The latter does not rely on this pipeline; however, it lacks the ability to learn object representations causing blurriness and object disappearances for longer-term predictions which forms a safety hazard. This thesis proposes two methods to improve the performance of the object-agnostic sequence-to-sequence Occupancy Grid Map (OGM) prediction networks, trained on the Waymo Open Perception dataset. The first method uses inter-pixel loss functions, i.e. the SSIM and Sinkhorn losses, instead of the ubiquitously used per-pixel losses, to train the PredRNN++ Occupancy Grid Map (OGM) prediction network. Inter-pixel losses take into account the spatial relations between grid cells during the evaluation of OGMs, whereas per-pixel losses evaluate each grid cell’s value independently. The quantitative results demonstrate that using inter-pixel losses can improve short term predictions with a prediction horizon of T = 5 for the Mean Squared Error (MSE) by 4.3%, Image Similarity (IS) by 7.8%, Average Precision (AP) by 0.3%, metrics. For the longer term, T = 15, the predictions improve for the MSE by 5.0%, IS by 20.5%, AP by 0.1%, and Accuracy by 0.6%. Furthermore, the use of inter-pixel losses reduces blurriness and object disappearances. The second method is based on multi-task learning. By training the PredRNN++ to perform the prediction task together with the semantic segmentation task on the predicted OGMs, it is expected to learn object representations which it uses to improve the prediction quality. The quantitative results show that multi-task learning does not improve the OGM predictions. However, some qualitative results show that multi-task learning reduces blurriness and object disappearances.

Self-driving vehicles have shown rapid development in recent years and continue to move towards full autonomy. For high or full automation, self-driving vehicles will have to be able to address and solve a broad range of situations, one of which is interaction with traffic agents. For correct and save maneuvering through these situations, reliable detection of agents followed by an accurate classification of the traffic gestures used by agents is essential. This problem has received limited attention in literature to date. The objective of this work is to establish and investigate a working traffic gesture pipeline by leveraging the latest developments in the fields of computer vision and machine learning. This work investigates and compares how well state-of-the-art methods translate to traffic gesture recognition and what application specific problems are encountered. Multiple configurations based on skeletal features, estimated using OpenPose, and classified using recurrent neural networks (RNN) were investigated. Skeleton estimation using OpenPose and feature representations were evaluated using an action recognition dataset with motion capture ground-truth. Three RNN network architectures, varying in complexity and size, were evaluated on traffic gestures. The robustness of the developed system regarding viewpoint variation is explored, combined with the viability of transfer learning for traffic gestures. To train and validate these methods, a new traffic gesture dataset is introduced, on which an mAP of 0,70 is achieved. The results show that the proposed methods are able to classify traffic gestures within reasonable computation time and illustrate the value of transfer learning for gesture recognition. These promising results validate the methodology used and show that this direction warrants further research.

The bottleneck of the maximum road volume in urban areas is the maximum capacity of the traffic flow on the intersection, which is coordinated with Traffic Light Controllers (TLCs). A promising method to decrease the number of stops are Green Light Optimal Speed Advice (GLOSA) systems. These systems will give a speed advice to arriving vehicles based on the schedule of TLCs, which needs to be known and fixed. However, most on-street controllers change their schedule until the last moment to maximize the performance. In this thesis a predictive controller is developed that is suitable for real-world application based on DIRECTOR; a state-of-the-art predictive controller. A prediction model is used to predict future arrivals based on available measurements to optimize and fix the schedule in advance. The proposed controller can enable GLOSA systems to improve performance. Appropriate pre-processing steps are implemented and the optimal input features are selected to improve the performance of a Long Short-Term Memory (LSTM) network to predict future arrivals. All detection data is stationary over time by using the differenced series. The time data is divided into workdays and weekend days to create a binary input and undesirable jumps during midnight are removed. The combination of stop line detectors, queue detectors, arrival detectors and signal states of the controlled and preceding intersections as input maximized the performance. The prediction horizon of the proposed prediction model could be extended. The Normalized Root Mean Square Error (NRMSE) decreased with 17% compared to DIRECTOR. The proposed controller extends the control horizon and uses multiple prediction models to predict the arrivals for the entire control horizon. The proposed controller outperforms DIRECTOR with 14 - 38% reduction in terms of vehicle delay and 5 - 32% reduction in terms of numbers of stops based on the scheduling mode. The GLOSA system is an add-on of the controller and is able to operate without the GLOSA system. The control horizon of the proposed controller always has a fixed length which is needed to determine the time until green. The implemented GLOSA system will determine the optimal speed based on the time until green and the expected delay due to the surrounding vehicles. The proposed controller is a cloud controlled application. Therefore, it is possible to adjust the setup (i.e. scheduling modes) during the day. Enabling GLOSA all day except during rush hours will lead to 3 - 4% reduction in terms of vehicle delay and 29 - 32% reduction in terms of numbers of stops based on the scheduling mode. This setup of the proposed controller is also competitive with the hand-crafted non-predictive on-street controller. Compared with this controller, the proposed controller will reduce the number of stops with 26% at the cost of 16% increase in vehicle delay. The proposed controller is designed conform the safety standards used on-street. Permission is received from the provincial government to do on-street pilots with the proposed controller. These results could give new insights into the performance of the proposed controller.

Passive acoustic sensing utilizes the ability of sound to travel beyond the line-of-sight to understand the surroundings. This provides an advantage over the currently used sensors in Intelligent Vehicles that can sense obstacles within their line-of-sight only. Recently, a localization based approach has been implemented to take advantage of this sensing modality to predict approaching vehicles behind the blind corner in an urban scenario. While this approach shows a lot of promise, there is a difficulty in integrating the multi-microphone system. Additionally, the system would be unable to differentiate between the nature of two sound sources. This motivates the exploration of a classification based approach which uses audio data from only a single microphone to identify the sound sources present in them. This thesis investigates the possibility of having such a system on the Intelligent Vehicle to predict approaching vehicles from behind the blind corners. A review of the literature revealed that techniques categorized under Sound Event Detection (SED) are suitable to implement a classification based approach. The prediction of the vehicle is treated as a binary classification problem and a Convolutional Recurrent Neural Network (CRNN) is used as the acoustic model to detect the presence of an approaching car in the audio sample represented by Log Mel Spectrogram features. Additionally, domain adaptation techniques were implemented to explore the possibility
of improving the system performance with limited data collected while the ego-vehicle is driving. Experiments carried out indicate that when the ego-vehicle is static, the system performs well with the approaching vehicle predicted 1.4s before it is in line-of-sight and a balanced accuracy of 86.9% achieved for the classification task. However, the system achieved an accuracy of 68% on the samples recorded while the ego-vehicle was driving. Further experiments indicate that the acoustic model cannot generalize well to unseen situations in most cases and experiment with domain adaptation did not show
any improvement in performance.

Human pose estimation, a challenging computer vision task of estimating various human body joints' locations, has a wide range of applications such as pedestrian tracking for autonomous cars, baby monitoring, video surveillance, human action recognition, virtual reality, gaming, gait analysis, etc. A majority of the research on the development of models for the task of human pose estimation has been focused on improving the accuracy of the task which also increases the complexity of the models. These models demand devices with high computational power to be deployed for real-world applications. Even though a lot of research has been focused on estimating the human pose from monocular images taken from cameras, the complexity of the models makes them impossible to be implemented on edge devices and embedded devices like mobile phones that have built-in cameras. This reduces the scope of applications where human pose estimation can be used. To address the issue, the research focuses on improving the performance of a baseline human pose estimation architecture by reducing the model size(number of parameters) and thereby its inference time without a significant loss in the accuracy. To improve the performance of the model, a structured Bayesian compression algorithm is used and the network is compressed by engineering the model based on the uncertainty of the parameters. The results show that the Bayesian compression method reduces the model size by around 65 percent with only a very little drop in the model accuracy. Also, the comparison of the inference time of the original baseline and the compressed model in an android device shows that the inference time is reduced by around 50 percent because of the reduction in the number of operations in the compressed model architecture.

Simultaneous Localization And Mapping (SLAM) algorithms provide accurate localization for autonomous vehicles and provide essential information for the path planning module. However, SLAM algorithms as- sume a static environment in order to estimate a location. This assumption influences the pose estimation in dynamic urban environments. The impact of this assumption on day-to-day scenarios of an intelligent vehicle is unknown. A deeper understanding on the effect of dynamic scenarios in an urban environment could lead to simple and robust solutions for SLAM algorithms in intelligent vehicles. The objective of this research is to develop a methodology that isolates the effect of an urban dynamic environment on the per- formance of a SLAM algorithm. This requires constant environment conditions including constant weather conditions, lighting conditions and identical trajectories over time. The methodology is tested with a stereo feature based V-SLAM algorithm called ORB SLAM [19], which illustrates the in-depth analysis that is possi- ble with this experiment. The main research question is: How does a dynamic urban environment influence the pose estimation accuracy of stereo ORB SLAM? Two specific dynamic scenarios are designed to represent a dynamic urban environment: driving behind another vehicle and vehicles approaching on the other side of the road. On these scenarios, an in-depth anal- ysis of ORB SLAM is performed to observe how the algorithm’s design influences the robustness to a dynamic environment. Functions within the algorithm are bypassed to analyze the effect on the performance. Specifi- cally, the place recognition function and map point filtering function are bypassed. The analysis proofs which functions assist in the overall robustness to a dynamic environment. Moreover, an analysis is performed of the algorithm in localization mode to research the effect of utilizing maps that were created under different conditions. The knowledge gained from the full analysis can be utilized to improve other V-SLAM algorithms. The experiment is performed in CARLA [6], an open source simulator. CARLA provides an elaborate sen- sor suite which support multiple camera setups and LIDAR sensors. Furthermore, the simulator provides free maps which represent realistic urban environments and allows for easy and accurate access to the ground truth position. A setup is designed with the simulator that allows complete isolation of the effect of a dy- namic environment. The setup allows full control of lighting conditions, weather conditions and allows iden- tical trajectories over time in different dynamic scenarios. Each scenario is simulated over several different trajectories in which the camera images are converted to rosbags. Each variation of the ORB SLAM algorithm is tested on the produced rosbags. The resulting pose estimations in dynamic conditions are compared to the pose estimations made during static conditions to analyze the effect of dynamic scenarios on the perfor- mance of the algorithm. The method successfully isolated the effect of a dynamic environment on the performance of stereo ORB SLAM. It allows for a detailed analysis which aids in finding the source of performance differences. In general, stereo ORB SLAM displays robust behavior to a dynamic environment. The experiment shows that the algo- rithm is sensitive to false relocalization when the stereo camera setup is driving 10 meters behind another vehicle for a long period of time. During these conditions, ORB SLAM cannot provide accurate pose esti- mations even when the place recognition module is deactivated. Furthermore, the map point filtering does increase the robustness in certain dynamic scenarios. Finally, the data suggests that utilizing maps created in different conditions does influence the pose estimation in localization mode. However, more data is needed to confirm these results. The methodology has proven its value for in depth analysis of robustness to an urban dynamic environ- ment for a SLAM algorithm. This experiment is not limited to ORB SLAM but could be utilized for other monocular and stereo V-SLAM methods, as well as LIDAR based methods. New solutions can be developed to increase robustness to a dynamic environment and tested on the same rosbags. This methodology could be an important tool for the development SLAM algorithms for intelligent vehicles.

One-Shot Neural Architecture Search (NAS) is a promising method to significantly reduce search time without any separate training. It can be treated as a Network Compression problem on the architecture parameters from an overparameterized network. However, there are two issues associated with most one-shot NAS methods. First, dependencies between a node and its predecessors and successors are often disregarded which result in improper treatment over zero operations. Second, architecture parameters pruning based on their magnitude is questionable. In this thesis, classic Bayesian learning approach is applied to alleviate these two issues. Unlike other NAS methods, we train the over-parameterized network for only one epoch before update network architecture. Impressively, this enabled us to find the optimal architecture in both proxy and proxyless tasks on CIFAR-10 within only 0.2 GPU days using a single GPU. As a byproduct, our approach can be transferred directly to convolutional neural networks compression by enforcing structural sparsity that is able to achieve extremely sparse networks without accuracy deterioration.

Deep learning methods have been demonstrated to be promising in semantic segmentation of point clouds. Existing works focus on extracting informative local features based on individual points and their local neighborhood. They lack consideration of the general structures and latent contextual relations of underlying shapes among points. To this end, we design geometric priors to encode contextual relations of underlying shapes between corresponding point pairs. Geometric prior convolution operator is proposed to explicitly incorporate the contextual relations into the computation. Then, GP-net, which contains geometric prior convolution and a backbone network is constructed. Our experiments show that the performance of our backbone network can be improved by up to 6.9 percent in terms of mean Intersection over Union (mIoU) with the help of geometric prior convolution. We also analyze different design options of geometric prior convolution and GP-net. The GP-net has been tested on the Paris and Lille 3D benchmark, and it achieves the state-of-the-art performance of 74.7 % mIoU.

Recent advances in sensor technology have lead to increased resolution of novel sensors, while tracking applications where distance between sensors and objects of interest is very small have gained research interest recently. In these cases, it is possible that multiple sensor detections are generated by each object of interest. Extended Object Tracking (EOT) approaches consist of algorithms which make use of multiple sensor detections per object to jointly estimate their kinematic and shape extent attributes within the Bayesian tracking framework. In the last decade, various EOT algorithms have been proposed for different types of tracking applications. This M.Sc. thesis project addresses the problem of extended tracking of a single pedestrian walking in the area of a stationary vehicle (referred as ego-vehicle in this report) during a real automotive scenario. The objective is to achieve accurate estimation of both the kinematic attributes (2D centroid position/velocity), as well as its shape extent in x-y plane. In more detail, PreScan software is enabled to design a simulation scenario that is very close to a real automotive application, in terms of motion characteristics of objects of interest and sensor data acquisition. In the considered scenario, different sensor modalities are mounted on the ego-vehicle, namely a Lidar sensor and a mono camera sensor. Moreover, OpenPose library is employed to to obtain pose detections of human body parts from obtained camera images. Concerning shape extent representation, the simplest and most popular approach in previous studies, in general and especially for VRUs tracking, is to assume an elliptical shape. In fact, the Random Matrix Model (RMM), proposed originally by Koch, 2008, is a state-of-the-art EOT state modeling approach that allows for joint estimation of centroid kinematics and physical extent for considered elliptical objects of interest. Based on that, a RMM-based filter using Lidar position measurements has been proposed by Feldmann, 2011. In this project, this algorithm is used as a baseline filter for comparison with our proposed algorithm.
In addition, an alternative tracking algorithm is proposed in this study, which has the following differences with respect to the baseline filter:
State Initialization of the filter: In our proposed version of the tracking algorithm, human pose detections of shoulders and ankles are are associated with obtained Lidar position measurements in order to provide initial values for the kinematic state (2D position/velocity) and shape parameters (ellipse orientation and semi-axes lengths) of the pedestrian.Measurement Update step of the filter: In our proposed version of the tracking algorithm, camera-obtained pose detections of pedestrian shoulders are associated with obtained Lidar position measurements in order to create an extra measurement, for pedestrian heading angle. Subsequently, a nonlinear filtering update step fusing Lidar-obtained point cloud data for pedestrian position and human-pose-obtained measurement for pedestrian heading angle is implemented. Both considered tracking algorithms are evaluated for the designed simulation scenario. In detail, the following performance metrics are used for evaluation of each filter: RMSE for estimated pedestrian 2D position and velocity, respectively.
Modified Hausdorff distance for estimated pedestrian shape extent.
In more detail, Monte Carlo simulations with multiple runs are designed to evaluate performance of each state initialization approach and each tracking algorithm, where the following parameters change in each run:
Additive zero-mean Gaussian measurement noise on obtained Lidar position detections. Initial simulation timestep.

Autonomous driving is a development that has gained a lot of attention lately, because it can lead to major improvements in the mobility sector. One example of a research project that aims to develop vehicles that are capable of reaching the highest level of autonomy in driving, is the WEpods project. The goal of this research is in line with this aim, having the thesis objective defined as follows: let the WEpods continue driving in autonomous mode more often than is currently the case.

The WEpod shuttles are not yet completely able to drive autonomously due to their inability to handle unexpected behavior (terminology: faults). Currently, such faults need to be detected and solved by a steward, who will manually initiate a safe stop if necessary. The localization module, which is responsible for localizing the vehicle on a map, sometimes generates unreliable location estimates. This poses two challenges. First, the fact that there is a mismatch between reality and the sensor outcomes of the localization module that needs to be detected. Second, the question of how to prevent the system from showing behavior that is different from what is desired (terminology: failure) in case such a fault is present (terminology: fault tolerant contol). Fault tolerant control can be performed in either a passive or an active manner. The passive approach ensures that either the faults are prevented or the system is able to mitigate them by anticipation in the design. The approach evolves from passive to active fault tolerant control when an on-line adaptation of the system control is made. For applications in autonomous driving, it is apparent that it is important to handle not only anticipated faults, but also to be able to deal with unexpected faults in an on-line manner. This on-line fault tolerant control approach involves two fault diagnosis steps that lead to solving the first challenge: detection and isolation.

A so-called model-based fault diagnosis approach turned out to be most suitable, as it has been used for similar applications in the past. However, a model-based fault diagnosis approach has not yet been implemented for detecting and isolating faults in a localization module of autonomous driving, indicating the scientific relevance of this research. In the model-based approach, kinematic and dynamic equations of the research vehicle (WEpod) are used to build a computational model. This model is then subjected to an observer, that is able to compare the model outcomes with the actual measurements in an off-line way. A residual is drawn up by taking the difference between the model outcomes and the measurements. A threshold is computed based on noise on the measurements to compare the residual with. When the residual exceeds the threshold, an alarm is raised. This way, the system itself has been enabled to detect faults when they occur internally.

Inclusion of the suggested fault diagnosis approach in an on-line manner into the system is a big step towards fully autonomous driving of the WEpods, and therefore the goal of this research is met.

A prevalent approach for learning a control policy in the model-free domain is by engaging Reinforcement Learning (RL). A well known disadvantage of RL is the necessity for extensive amounts of data for a suitable control policy. For systems that concern physical application, acquiring this vast amount of data might take an extraordinary amount of time. In contrast, humans have shown to be very efficient in detecting a suitable control policy for reference tracking problems. Employing this intuitive knowledge has proven to render model-free learning strategies suitable for physical applications. Recent studies have shown that learning a policy by directive action corrections is a very efficient approach in employing this do-main knowledge. Moreover, feedback based methods do not necessarily require expert knowledge on modelling and control and are therefore more generally applicable. The current state-of-the-art regarding directional feedback was introduced by Celemin and Ruiz-del Solar (2015) and coined COrrective Advice Communicated by Humans (COACH). In this framework the trainer is able to correct the observed actions by providing directive advise for iterative policy updates. However, COACH employs Radial Basis Function (RBF) networks, which limit the capabilities to apply the framework on higher dimensional problems due to an infeasible tuning process.This study introduces Gaussian Process Coach (GPC), an algorithm preserving COACH’s structure, but introducing Gaussian Processes (GPS) as alternative to RBF networks. Moreover, the employment of GPS allows for uncertainty estimation of the policy, which will be used for 1) inquiringhigh-informative feedback samples in an Active Learning (AL) framework, 2) introduce an Adaptive Learning Rate (ALR) that adapts the learning rate to the coarse or refine focused learning phase of the trainer and 3) establish a novel sparsification technique that is specifically designed for iterative GP policy updates. We will show by employing synthesized and human teachers that the novel algorithm outperforms COACH on every domain tested, with the most outspoken difference on higher dimensional problems. Furthermore, we will prove the independent contributions of AL and ALR.

This work explores the possibility of incorporating depth information into a deep neural network to improve accuracy of RGB instance segmentation. The baseline of this work is semantic instance segmentation with discriminative loss function.The baseline work proposes a novel discriminative loss function with which the semantic net-work can learn a n-D embedding for all pixels belonging to instances. Embeddings of the same instances are attracted to their own centers while centers of different instance embeddings repulse each other. Two limitations are set for attraction and repulsion, namely the in-margin and out-margin. A post-processing procedure (clustering) is required to infer instance indices from embeddings with an important parameter bandwidth, the threshold for clustering. The contribution of the work in this thesis are several new methods to incorporate depth information into the baseline work. One simple method is adding scaled depth directly to RGB embeddings, which is named as scaling. Through theorizing and experiments, this work also proposes that depth pixels can be encoded into 1-D embeddings with the same discriminative loss function and combined with RGB embeddings. Explored combination methods are fusion and concatenation. Additionally, two depth pre-processing methods are proposed, replication and coloring. From the experimental result, both scaling and fusion lead to significant improvements over baseline work while concatenation contributes more to classes with lots of similarities.

This work presents a multi-sensor approach for weather condition estimation in automated vehicles. Using combined data from weather sensors (barometer, hygrometer, etc) and an in-vehicle camera, a machine learning and computer vision framework is employed to estimate the current weather condition in realtime and in-vehicle. The use of different sensor types is shown to improve robustness and reduce noise. The resulting modular framework allows it to be used with different sensor configurations, and allows changes in sensor configuration with minimal effort. Finally, a proof-of-concept experiment is presented; a dataset is recorded using a test vehicle and used for model evaluation.
The resulting datasets contains 20.000 pairs of video frames and sensor measurements recorded in different weather situations.