Behavior-agnostic reinforcement learning is a rapidly expanding research area focusing on developing algorithms capable of learning effective policies without explicit knowledge of the environment's dynamics or specific behavior policies. It proposes robust techniques to perform off-policy evaluation, namely Distribution Correction Estimation (DICE) methods, in the context of infinite horizon Markov Decision Processes (MDPs). This research paper investigates the impact of the initial start distribution mismatch on the accuracy of DICE estimators in behavior-agnostic reinforcement learning. To achieve this, seven systematic initial start distributions were created and utilized to calculate the initial start distribution mismatch via Kullback–Leibler (KL) divergence. Furthermore, off-policy evaluation performance was assessed using DICE estimators, with Mean Squared Error (MSE) comparisons against ground truth values. The study reveals that, based on the conducted experiments, the initial start distribution mismatch does not have a clear influence on the performance of the DICE estimators. Therefore, future research is required to increase the scope of the experiments and address some of the limitations of this study to accurately assess the impact of the initial start distribution mismatch on off-policy evaluation using DICE methods. This paper underscores the complexity of the initial start distribution choice in behavior-agnostic reinforcement learning, calling for further research to effectively evaluate its impact across diverse environments and measures. Additionally, exploring the relation between the initial start distribution and policies could provide deeper insights and further refine the understanding of their influence on DICE estimators.

Off-policy evaluation has some key problems with one of them being the “curse of horizon”. With recent breakthroughs [1] [2], new estimators have emerged that utilise importance sampling of the individual state-action pairs and reward rather than over the whole trajectory. With the difference between behaviour and target policy, the state-visitation mismatch occurs. This paper is interested in answering the question how the degree of state-visitation mismatch affects the overall target policy performance. The approach is to calculate the state-visitation mismatch with the KL divergence, which consists of the state-visitation distribution of the behaviour policy and the distribution correction ratio of the DICE estimator. The state-visitation mismatch can be quantified in way. Furthermore, the effect on the target policy performance is quantified by the MSE between the estimated, empirical cumulative reward and the estimated reward by the DICE estimator. By analysing the KL divergence and MSE value, one may argue that the state-visitation mismatch does impact the performance of the target policy but further research needs to be conducted.

This paper addresses the issue of double-dipping in off-policy evaluation (OPE) in behaviour-agnostic reinforcement learning, where the same dataset is used for both training and estimation, leading to overfitting and inflated performance metrics especially for variance. We introduce SplitDICE, which incorporates sample-splitting and cross-fitting techniques to mitigate double-dipping effects in the DICE family of estimators. Focusing specifically on 2-fold and 5-fold cross-fitting strategies, the original off-policy dataset is partitioned with random-split to get separate training and evaluation datasets. Experimental results demonstrate that SplitDICE, particularly with 5-fold cross-fitting, significantly reduces error, bias, and variance compared to naive DICE implementations, providing a more doubly-robust solution for behavior-agnostic OPE.

In the field of reinforcement learning (RL), effectively leveraging behavior-agnostic data to train and evaluate policies without explicit knowledge of the behavior policies that generated the data is a significant challenge. This research investigates the impact of state visitation mismatch methods on the performance of on-policy RL methods, an area crucial for improving policy performance in real-world applications where behavior policies are often unknown. Specifically, we compare the convergence speed and performance of Q-learning when initialized with Q-values learned through the Distribution Correction Estimation (DICE) method versus traditional random initialization. By generating datasets representing behavior and target policies, we employ the DICE estimator to initialize Q-values, and subsequently run Q-learning for both DICE-initialized and randomly-initialized scenarios. Our results demonstrate that initializing Q-learning with DICE Q-values enhances convergence speed, leading to faster attainment of near-optimal policies. This study provides valuable insights into the effectiveness of state visitation mismatch methods in improving the efficiency and performance of on-policy RL algorithms, contributing to the development of more robust RL applications in behavior-agnostic settings.

In offline reinforcement learning, deriving a policy from a pre-collected set of experiences is challenging due to the limited sample size and the mismatched state-action distribution between the target policy and the behavioral policy that generated the data. Learning a dynamic model of the environment can improve the sample efficiency of the algorithm, but this mismatch can lead to the generation of suboptimal experiences. We propose SimuDICE, an algorithm that enhances the sampling of imaginary experiences using Dual stationary DIstribution Correction (DICE), and iteratively improves the DICE estimations with synthetically generated experiences. SimuDICE addresses the objective mismatch issue by iteratively updating both the world model and the DICE estimator, aligning the model's training objective (imitating the environment) with its usage objective (policy improvement). We show that SimuDICE requires less pre-collected data and fewer simulated experiences to achieve comparable results to other algorithms while having greater robustness to lower data quality.

In order to develop artificial agents that can understand social interactions at a near-human level, it is required that these agents develop an artificial Theory of Mind; the ability to infer the mental state of others. However, developing this artificial Theory of Mind is a highly difficult process. This is because Theory of Mind is an ambiguous and multifaceted concept, having several mechanisms associated with it, and being tested using many different tasks. In this thesis, we formalize what mechanisms constitute Theory of Mind, and establish how we can represent these mechanisms using artificial intelligence. Furthermore, we evaluate whether current artificial Theory of Mind models are able to reason effectively about these mechanisms. This is done by creating Theory of Mind tasks for artificial models, evaluating their effectiveness, and allowing us to provide recommendations for the development of future artificial Theory of Mind models.

Spectral Monte-Carlo rendering can simulate advanced light phenomena (e.g., dispersion, caustics, or iridescence), but require significantly more samples compared to trichromatic rendering to obtain noise-free images. Therefore, its progressive variant typically exhibits an extreme amount of chromatic noise in early renders. To that end, we propose a two-stage progressive approach. We initially restrict the original wavelength distribution, then slowly relax it. In the process of relaxing the range of wavelengths, all wavelengths that are outside of that restricted range will be propagated. Thereby, we lower variance and increase the perception of these early renders with little overhead.

Indirect illumination is an essential part of realistic computer-generated imagery. However, accurate calculation of indirect illumination comes at high compute costs. To this end, we replace lengthy indirect illumination paths by employing an ambient light cache based on photon mapping principles. By only performing cache queries after a certain path length has been reached, we show that significantly fewer photons than traditional photon mapping techniques are sufficient and that an inaccurate data structure can be used to store them. Despite these simplifications, our method generally outperforms unidirectional path tracing while adding little time overhead.

Spectral Monte-Carlo methods are powerful physically-based techniques for simulating wavelength-dependent phenomena such as dispersion. However, compared to tristimulus rendering, they involve sampling the spectral domain, which adds substantial overhead, requiring significantly more samples for noise-free, realistic-looking renders. Thereby, we propose a simple approach to efficiently sample emitters. We precompute a simple 2-dimensional data structure using spectral power distributions of scene emitters. We use it to model a probability distribution function to sample an emitter that yields high path throughput at every intersection when using Next Event Estimation. Our method handles various geometries and spectral distributions of scene emitters, improves convergence, and reduces noise with negligible overhead.

Path tracing is a well-known light transport algorithm used to render photo-realistic images. However, it is an expensive algorithm with an active area of research for improving its efficiency. In our work, we present a method to measure and visualize the regions of high computational cost for unidirectional path tracers. We have defined metrics to estimate the cost per pixel that can be visualized as a two-dimensional(2D) image. To show its usefulness, we present how our method can visualize the effect of changing material properties, object placement and other scene variables on computational cost. With this insight, the user can make clever choices to improve computational time.

Direct lighting calculation is an essential part of photorealistic rendering. Standard importance sampling techniques converge slowly in scenes where a light source is only visible through small openings as visibility is not considered. This problem is often addressed by manually placing light portals, marking the openings to the light. However, existing portal sampling techniques are not suitable for area lights since the portal is often times larger than the light itself. We present a novel portal sampling technique inspired by shadow volumes, which considers the light geometry to efficiently choose the optimal sampling strategy, depending on the location of the shading point. Our technique is unbiased, robust, applicable to many scenes, and easy to integrate into an existing renderer.

Collaboration in teams composed of both humans and automations has an interdependent nature, which demands calibrated trust among all the teammembers. For building suitable autonomous teammates, we need to study how trust and trustworthiness function in such teams. In particular, automations occasionally fail to do their job, which leads to a decrease in human’s trust. However, research has given contradictory statements about the effects of such a reduction of trust on the human’s trustworthiness, i.e. human’s characteristics that make them more or less reliable to the automation. As such, this study investigates how automation failure in a human-automation teamwork scenario affects the human’s trust in the automation and human’s trustworthiness towards the automation. We present a between-subjects controlled experiment in which the participants perform a simulated task in a 2D grid-world, collaborating with an automation in a “moving-out” scenario. During the experiment, we measure the participants’ trust and trustworthiness regarding the automation both subjectively and objectively. Our results show that automation failure negatively affects the human’s trustworthiness, as well as their trust in and liking of the automation. Learning the effects of automation failure in trust and trustworthiness can contribute to a better understanding of the nature and dynamics of trust in these teams, foreseeing undesirable consequences and improving human-automation teamwork.

Adversarial attacks pose a risk to machine learning (ML)-based network intrusion detection systems (NIDS). In this manner, it is of great significance to explore to what degree these methods can be viably utilized by potential adversaries. The majority of adversarial techniques are designed for unconstrained domains such as the image recognition domain, where these methods apply alterations to the pixels in a picture. Therefore, the applicability of these techniques to the NIDS domain is very limited. Related work on adversarial techniques for NIDS generally considers feature-space techniques, which cannot be applied in a practical situation since only the extracted network traffic features are modified and not the actual network traffic. To solve these limitations, a traffic-space approach for creating adversarial examples for evading ML-based NIDS is proposed and assessed with several classification models. The proposed constrained adversarial crafting method is based on the Iterative Fast Gradient Sign Method (IFGSM) and is called the Constrained Iterative Fast Gradient Sign Method (CIFGSM). A constraint set is added as a penalty term to the loss function of the optimization to ensure that the adversarial values remain within the valid space. Additionally, an L2 regularization term is used to minimize the distance between the original and adversarial network traffic samples. The proposed method is evaluated and shown to be an effective way for generating realistic and practical adversarial evasion packets. To achieve this, network packet components and their characteristics are defined as a constraint set which can be used for the optimization task and a custom adversarial loss function is created that encapsulates the different elements of this optimization problem. Furthermore, multiple models are evaluated to test the transferability of this method. Conclusively, the proposed method is evaluated in a realistic scenario, where adversarial packet captures are crafted and examined. Where other state-of-the art works only modify the network traffic features in feature-space or on a connection level only and do not apply their method in a real world scenario, this work modifies the packet captures on a per-packet level which is subsequently used to evaluate flow based classification models.

This research studies the Projected Bidirectional Long Short-Term Memory Time Delayed Neural Network (TDNN-BLSTM) model for English phoneme recognition. It contributes to the field of phoneme recognition by analyzing the performance of the TDNN-BLSTM model based on the TIMIT corpus and the Buckeye corpus, respectively containing read speech and spontaneous speech. The TIMIT corpus can be used as benchmark to make comparisons between architectures. The Buckeye corpus is used to better understand how the TDNN-BLSTM architecture would perform on recorded informal conversations.
Parameter values are taken from literature and are optimized.
Using the improved parameters, the results show Phoneme Error Rates (PER) for read speech to be 31.78% and for spontaneous speech to be 54.03%. Related work shows PER scores for read speech to be 14.9% and for spontaneous speech to be 23.4%.
This indicates that the TDNN-BLSTM architecture does not perform as well as other acoustic models for both spontaneous and read speech.

This research expands past research on implementing the TDNN-OPGRU network for Automatic Phoneme Recognition on Dutch speech by implementing and testing the TDNN-OPGRU network on Mandarin speech. The goal of this research is to investigate the performance of the TDNN-OPGRU architecture when decoding phonemes in Mandarin prepared and spontaneous speech. The difference in Phoneme Error Rate between prepared and spontaneous speech is being determined, and the effect that tones have on the PER is being investigated since Mandarin is a tonal language. The results are that a substantial amount of the PER comes from substitutions that are made where only the tone is incorrectly determined. However, tone does not appear to have an impact on the difference in PER between spontaneous and prepared speech since it is responsible for an similar amount of the substitutions in both types of speech. The inclusion of tone also causes the error rate of the TDNN-OPGRU architecture on base phonemes to increase.

A limitation of current ASR systems is the so-called out-of-vocabulary words. The solution to overcome this limitation is to use APR systems. Previous research on Dutch APR systems identified Time Delayed Bidirectional Long-Short Term Memory Neural Network (TDNN-BLSTM) as one of best performing state-of-the-art NN architecture for PR. The goal of this research is to evaluate the performance of the TDNN-BLSTM architecture for phoneme recognition on Mandarin read and spontaneous speech, analyze the differences in performance for the two speech styles as well as compare the results with previous research on Dutch PR.

To achieve this goal 4 different NN models of the TDNN-BLSTM architecture were built and trained on Mandarin read and spontaneous speech. The test results of the NN models were used to calculate the phoneme error rate (PER), decomposed PER, and the contribution of individual phonemes to the overall PER. Based on these findings, conclusions are formulated regarding the impact of different languages, speech styles, and the architectural changes on the performance of the TDNN-BLSTM architecture.

In September 2018, the Smart Teddy project was founded by a group of researchers within the Hague University of Applied Sciences1 in the Netherlands. The Smart Teddy project is a multidisciplinary project aiming to create an interactive system, using a teddy bear as a focus point, which collects the data needed in order to enable seniors with dementia to live independently for a longer period of time. Over the last three years, three prototypes of the Smart Teddy have been developed. The Smart Teddy project was introduced as a final project for students following the BSc program Electrical Engineering at the Delft University of Technology. Starting in April 2021, a team of six students attempted to further develop the Smart Teddy over the course of 11 weeks. This thesis contains the Human Interaction & Integration subdomain of the Smart Teddy thesis project, where Human Interaction refers to the aspects of the Teddy that encourage interaction with the user, and Integration refers to the combination of all subdomains into one fully functioning prototype. In this thesis, the design choices, implementation methods and verification are discussed. The contribution to the prototype regarding Human Interaction & Integration are the addition of a movement system using pneumatics, the implementation of a flexible touch sensor, the ability for the Teddy to produce audio, to communicate wirelessly with the Base Station, and for all components in the Teddy to communicate with the main controller. The final prototype has been implemented using the Raspberry Pi Pico microcontroller, which was mounted on a custom PCB. All controls are provided by the Pico, and uses I2C, SPI, UART, and analog and digital inputs to communicate with the sensors and actuators. These sensors and actuators were implemented using off-the-shelf breakout boards and drivers, to allow for fast design and test iterations. The movement of the Teddy has been implemented using air pumps and molded silicon rubber, and the tail wagging is implemented using the same principle used for soft robotic grippers. The final prototype is fully functional and meets 16 of the 20 requirements - the requirement concerning speech recognition and the noise produced by the pumps have not been met.

The amount of people dealing with dementia is rising globally. The amount of caretakers is, however, not. Therefore, technological aids are needed to support people dealing with dementia and relieve the stress on their caretakers. Current solutions provide tracking of people with dementia. Also, different robots exist that provide people with companionship. However, no solution exists that combines tracking and companionship capabilities. Therefore, the Smart Teddy is introduced. The Smart Teddy can track different indicators that indicate the progress of dementia and simultaneously provide the user with companionship through interaction. The goal of this thesis is to design a data acquisition system that acquires meaningful data that can be used for the development of algorithms that will autonomously determine the progress of dementia. To achieve this, a system with a Teddy and a Base station has been designed. The Teddy has a sound-, a carbon monoxide-, a smoke- and a movement sensor. Also, a real-time module is implemented to be able to assign the current time to the measurement data. Lastly, a GPS and GSM module is implemented to be able to track seniors in case they wander. In the Base station, a mmWave sensor is implemented that tracks the position, velocity, and direction of the persons present in the room. Also, a processor is implemented that gathers and stores the data from the mmWave sensor and the data from the Teddy which is sent via a LoRa connection. In addition, the designed system can store the collected data for more than one week. The collected data can be used by an expert in dementia to extract meaningful information about dementia progress, after that, an expert in digital signal processing is needed to develop algorithms that estimate the quality of life of a senior suffering from dementia.

The Decentralized Partially Observable Markov Decision Process is a commonly used framework to formally model scenarios in which multiple agents must collaborate using local information. A key difficulty in a Dec-POMDP is that in order to coordinate successfully, an agent must decide on actions not only using its own information, but also by reasoning about the information available to the other agents. Nevertheless, existing value-based Reinforcement Learning techniques for Dec-POMDPs typically take the individual perspective, under which each agent optimizes its own actions using solely its local information, thus essentially neglecting the presence of others. As a result, the concatenation of individual policies learned in this way has a tendency to result in a sub-optimal joint policy. In this work, we propose to additionally condition such Independent Q-Learners on the plan-time sufficient statistic for Dec-POMDPs, which contains a distribution over the joint action-observation history. Using this, the agents can accurately reason about the resulting actions the other agents will take, and adjust their own behavior accordingly. Our main contributions are threefold. (1) We thoroughly investigate the effects of conditioning Independent Q-Learners on the sufficient statistic for Dec-POMDPs. (2) We identify novel exploration strategies that the agents can follow by conditioning on the sufficient statistic, as well as their implications on the decision rules, the sufficient statistic and the learning process. (3) We substantiate and demonstrate that by conceptually sequencing the decision-making, and additionally conditioning the agents on the current decision rules of the earlier agents, such learners are able to consistently escape sub-optimal equilibria and learn the optimal policy in our test environment, Dec-Tiger.

Personality modeling is important in order to create character variation in
games. Character variation favors replayability and is an important aspect
of game design. The eect of articial personalities through the expression of
emotions is evaluated in this research. To do so, a prototype game is developed
in the context of training in bad news conversations. Replayability through
character modeling is important in a game in which people can train to deliver
bad news. By training with multiple personalities one can learn to deal with
the dierent reactions that people can give. In this thesis, the eect of arti-
cial personalities through the expression of emotions on the replayability of the
game, believability of the non-playing character and immersion of the player is
researched. It is expected that articial personalities have a positive eect on
the replayability of the game and believability of the non-playing character, but
not on the immersion of the player. Experiments show that there is a positive
trend in the replayability of the game and the believability of the non-playing
character.