Visual impairment affects over 2.2 billion individuals globally, emphasizing the critical need for effective assistive technologies. This work focuses on developing a video captioning model explicitly tailored for visually impaired users, leveraging advancements in deep learning techniques. Video captioning involves converting video frames into textual descriptions, effectively bridging the domains of computer vision (CV) and natural language processing (NLP). We surveyed young visually impaired individuals from the Visio organization, who provided key insights into the design of our model.
We enhance the existing S2VT model by modifying its temporal attention mechanism to improve the recognition of visual surroundings, addressing the unique challenges visually impaired individuals face.
This research explores critical questions surrounding the model's sensitivity to actions, the readability of generated captions, and methods for latency reduction. To evaluate the model's effectiveness, we implement readability metrics—an approach not previously utilized in video captioning assessments. Our findings contribute to enhancing accessibility and independence for visually impaired individuals through advanced video captioning solutions.

Moral values influence humans in decision-making. Pluralist moral philosophers argue that human morality can be represented by a finite number of moral values, respecting the differences in moral views. Recent advancements in NLP show that language models retain a discernible level of knowledge in deontological ethics and moral norms of society. However, a model which can only decide either right or wrong cannot fully understand the diverse moral perspectives of humans.

We propose a moral sentence embedding space, which can encompass moral differences, through the state-of-the-art Contrastive Learning framework. We evaluate the moral embedding space both intrinsically and extrinsically via three tasks: classification, moral similarity, and visual analysis. We show that our moral embedding space understands the characteristics of each moral value. Our results also highlight that moral rhetoric is seldom explicit in the text, emphasizing the necessity of additional information such as moral labels.

Training deep learning models for time-series prediction of a target population often requires a substantial amount of training data, which may not be readily available. This work addresses the challenge of leveraging multiple related sources of time series data in the same feature space to improve the prediction performance of a deep learning model for a target population. Specifically, we focus on a scenario where the target dataset, representing the desired target population, is underrepresented, while the source datasets consist of mismatched populations that are sufficiently representative for training a deep learning model. In this study, we explore state-of-the-art techniques, including transfer learning, ensemble learning, and domain adaptation to leverage source datasets towards a target population using real-world medical data. Additionally, we investigate the use of model performance-derived baselines as a heuristic to quantify the magnitude of the distribution mismatch between a source(s) and a target. Our results demonstrate that a set of well-defined baselines can effectively quantify the distribution mismatch and provide insights into the choice of leveraging technique for a given mismatch scenario. Furthermore, our results show that all state-of-the-art techniques can be employed to leverage related source datasets towards the target, though the performance of these techniques varies depending on the characteristics of the distribution mismatch. Eventually, we discuss the applicability of this research to new scenarios, along with avenues for future research.

The Artificial Social Agent(ASA) questionnaire serves as a tool to assess the interaction between humans and ASA\cite{OGQuestionnaire}. In an effort to increase the number of people who can make use of the questionnaire, previous work has been done to translate this questionnaire to Mandarin Chinese\cite{Chinese}. To increase the number of people who can use this questionnaire even more, we have translated it to German using a similar iterative translating process. We had professionals translate the questionnaire and ran 3 rounds of surveys to see what questions needed improvement. Then, there was a final survey that evaluated the German version to see how well it performs and to see how it compares to the English version. In order to evaluate the cultural differences between the Chinese and German translations, the data of both translation evaluations were compared. From these results, it was shown that ASA developers who use the German version of the ASA questionnaire can make comparisons on a construct level to the English version. This was, however, only possible when using the full questionnaire, not the short version. Additionally, we have found that there are differences in how Mandarin Chinese speakers and German speakers rate ASAs.

Artificial social agents (ASAs) are systems designed to interact with humans in a socially intelligent manner. As the field of robotics is rapidly advancing, some studies focused on creating more effective agents by analysing how people perceive them. However, culture affects people’s perception of ASAs. So, cultural aspects must be examined in order to create more effective ASAs. This study aims to contribute to the research of cultural influence on ASAs by discovering the cultural similarities and differences in the perception of ASAs from Dutch and Chinese speakers. An English questionnaire has been made to evaluate ASAs and also has been translated into Chinese. In this study, the questionnaire is translated into Dutch and validated using both the original and the translated questionnaire. While assessing the translation, data is gathered from Dutch speakers on a human-ASA interaction. This data is then compared to the previously collected data from the Chinese translation study by calculating their mean, standard deviation and t-distribution of the differences between both cultural groups. On an item level, the results show a satisfactory level of correlation (ICC M = 0.67, SD = 0.12, range [0.36, 0.92]). On a construct/dimension level, results show a good level of correlation (ICC M = 0.80, SD = 0.11, range [0.52, 0.93]). Correction values for the translation have been advised for converting item scores between the two questionnaires. Cultural differences have been found and reported between Dutch and Chinese speakers, which could be used in future research on creating more effective ASAs.

Humans interact with various Artificial Social Agents (ASAs) on a daily basis. ASAs range from the Honda robot ASIMO to Apple’s Siri. To measure the perception of human-ASA interactions, a standardized questionnaire was created. Yet, this questionnaire was so far only available in English and Chinese. It has been found that culture can affect how these interactions are perceived. The aim of this study is to answer the question: What are the differences and similarities of the English and German human-ASA interaction interpretations? In this paper, we translate the questionnaire into German, validate it. Once proven valid, we give the English and German questionnaire on bilingual participants who watch a human-ASA interaction video and rate it in both languages. We measure the differences and similarities between the English and German responses. At the end, we combine the finding from the questionnaire results with examples from literature to form recommendations for future ASA developments. We conclude that an average good level of correlation between the two languages for the 90 questionnaire items (ICC M = 0.65, SD = 0.14, range [0.27, 0.90]), on the construct level (ICC M = 0.8, SD = 0.1, range [0.51, 0.92]), and for the 24 representative items (M = 0.67, SD = 0.14, range[0.31, 0.90]). Additionally, we found systematic differences between the English questionnaire scores of the bilingual sample seen in this study and a previously established mixed-English sample.

The Artificial-Social-Agent (ASA) questionnaire is an instrument for rating Human-ASA interactions. The questionnaire was built to make evaluation results more comparable between different agents and to make findings more generalizable. The ASA questionnaire was designed in English, which limits its usage to people who have a firm grasp of the English language, so translating the questionnaire into different languages allows for the inclusion of other people and cultures, as well as opening up the possibility of comparing different cultures. A previous study performed a translation of the ASA questionnaire into Chinese, and this repeats a similar process by translating the questionnaire into Dutch. This research project employs a survey-based approach, utilizing the Prolific and Qualtrics platforms to collect valuable insights and responses from bilingual English and Dutch participants. Participants watch a video depicting an ASA-human interaction and then complete a questionnaire designed to assess their perceptions. The main research question focuses on identifying the cultural variations in the perceptions of ASAs. By addressing this question, this research contributes to the existing body of knowledge on ASAs and culture. The methodology involves translating and refining the questionnaire, followed by data collection and analysis using Intraclass correlation coefficients (ICCs) to measure agreement between the English and Dutch versions of the questionnaire. The final evaluation and analysis includes a deeper statistical analysis, and a comparison between Dutch-English bilingual participants, and mixed-English participants.

The work presented in this thesis investigates the creation of virtual sound sources in a room equipped with a limited number of loudspeakers. This limited number of loudspeakers is typical for consumer loudspeaker systems. Ideally, these systems can provide a listening experience in which localisation cues are completely present. However, in current systems, this is not the case. The limited number of loudspeakers make it impractical to produce a physically accurate sound field. A possible solution is to create a perceptually accurate sound field instead. In this work, a step towards an algorithm which can do so is presented. The developed algorithm requires knowledge of the listener placement, the room, and the loudspeaker placement. Spatial weighting is used to construct a region in which the acoustic en-ergy should be large and a region in which the acoustic energy should be limited. The loudspeaker playback signals are obtained by maximising the energy ratio between these regions, while at the same time ensuring that the perceptual difference between the received audio and reference audio remains limited. The algorithm employs a convex optimisation problem to facilitate efficiently solving for the playback signals. Concretely, six convex optimisation problems are proposed with somewhat increasing complexity and different weighting matrices. For each of the optimisation problems, the proposed algorithm is compared against a simple amplitude panning algorithm and a nearest neighbour algorithm. It is found that none of the considered algorithms is clearly preferred over the others in terms of the considered evaluation metrics. A major limitation of the presented work is that the evaluation metrics do not explicitly test for localisation accuracy. In future work, this should be investigated by including subjective tests.

Over the past decade, spatial audio awareness evolved into an in-demand feature in audio entertainment. The addition of sound source locations to, for instance, movies or music adds a level of auditory envelopment and spatial awareness to the audio experience. Expensive setups present in, for instance, cinema's, are able to create this envelopment by means of a large set of loudspeakers with which the desired sound fields are created. Creating this spatial envelopment in practical consumer living rooms or home cinema setups proves to be a more challenging task due to the impractical amount of loudspeakers required. To create the audio envelopment with a small amount of loudspeakers, crosstalk cancellation can be used. Crosstalk cancellation as posed in literature is, however, not robust enough to be used in practical appliances. The main cause of these bad characteristics is the objective cost function it optimizes which results in an ill-posed problem. In this thesis, the crosstalk cancellation problem is relaxed by aiming for perceptually sufficient results instead of aiming for objectively optimal results. The human auditory system has its limitations in both the perception of audio and localization of audio sources. Exploiting the limitations of the auditory system generates mathematical freedom that can be used to construct a more robust and stable crosstalk cancellation algorithm. This thesis provides the first steps towards the incorporation of audio perception in the domain of crosstalk cancellation and audio envelopment with a limited amount of loudspeakers.

We all know the possible consequences of global warming, rising temperatures, flooded cities and destroyed ecosystems. One of the causes is the emission of gases, predominantly CO2, which is increased by the growing E-commerce market. E-commerce companies rely on recommender systems to stimulate users to purchase products. We are convinced that we can use the core strength of recommender systems, influencing decision making, to steer users towards eco-friendly choices. Therefore, in this thesis, we research how greenness can be integrated into recommender systems. We present the first recommender system dataset that includes greenness, we benchmark several recommendation algorithms and we propose a strategy to increase recommendation greennness. To create the dataset, we annotate an existing recipe recommendation dataset with recipe greenness. For our benchmarking experiment, we propose metrics to measure recommendation greenness, which we use to show that no recommendation algorithm is fundamentally greener than others. Lastly, we propose a re-ranking method for improving the greenness of recommendation rankings. We use the method to explore the trade-off between accuracy and greenness and we show that it is possible improve the greenness of recommender systems significantly with little loss of accuracy.

The Socially Perceptive Computing Lab (SPCL) at Delft University of Technology has developed a device called the Midge. The aim of this device is to record data of social interactions at conferences. This paper aims to characterise how battery life is affected by different sensor settings on the Midge as well as how much data is generated in a given period of time. Based on the findings from the performed tests, formulae for both run time and data generation were devised.

The Midge is a wearable badge created by the Socially Perceptive Computing Lab, Pattern Recognition and Bioinformatics group of the Delft University of Technology, with as goal to analyse human behaviour. The badge has a digital motion processor (DMP) that can determine its orientation. This DMP makes use of an inertial measurement unit (IMU), that houses an accelerometer, a gyroscope and a magnetometer, to calculate its movement in a 3D-space. For both of the accelerometer and gyroscope the Full Scale Range (FSR) can be changed, in addition to the frequency. In this paper, the effects of both elements are analysed to determine if they influence the accuracy of the data gathered. The results show that the changing the FSR does not influence the accuracy of neither the two sensors nor the performance of the DMP. On the other hand, it was found that changing the frequency does influence the performance of the Midge. Even though the frequency did not affect the measurements of the accelerometer and gyroscope directly, the performance of the DMP was affected. The DMP performed best with a frequency of 150 Hz. Using a higher frequency also captured local extremes and turning points from the sensors and the interpreted orientation more precisely.

In our daily life people encounter many social interactions, for example in the supermarket, at work and in schools. Currently the most reliable way to find social interactions in groups, is to manually annotate the data. Manual annotation takes a lot of time and human resources and as the information stream goes faster and faster, the manual labour cannot keep up. Therefore an automated program to replace this is desirable. One method to automate this, is by looking at the proximity of people, which has been done by Dikker [1]. The results look promising, however they are sensitive to errors. The F1-score (equation 3) is only 0.625 and the precision (equation 5) is 0.696. This means that there are still a lot of false positives. With orientation data one can determine if the people in question are facing each other before they are assigned to be in social interaction. With a view frustum (cone shape) it can be determined who are facing at each other by checking if the cones overlap. This cone can take different sizes, making the view area bigger or smaller. Because the absolute rotation were only used, it was used to find out who cannot face each other. Using the proximity results as a baseline, the impossible orientations were removed. With the goal to reduce the amount of false positives. From the results it can be concluded that a small cone will perform worse compared to the baseline. The bigger the angle of the cone becomes, the closer it gets to the results from Dikker [1]. However, in none of the results it turns out that using absolute orientations improves the current work significantly.

Detecting social interactions through wireless wearable Bluetooth devices is increasing in popularity. Devices use the signal strength to other detected devices to estimate the proximity between people and group them together based on the Dominant set algorithm. Dominant sets are a maximal clique of nodes with an edge-weight based on the affinity between the nodes. Nevertheless, the signal is heavily influenced by external factors, which increase in an crowded environment. This paper introduces three different noise reduction filters that try to detect the kind of noise and therefore improve the detection of surrounded devices. Further, this paper looks at the overall impact of proximity on the resulting RSSI values. Knowing this relationship helps to normalize the values and therefore eliminates the need to apply noise reduction. Using a dataset of 48 sensors recorded in a conference setting with a specific designed sensor the low frequency pass filter gets an accuracy score of 81.8 % with a cut-off frequency of 0.07 Hz. It performs best when considering a conversation window of 20 seconds. Here, only 2/3 of the detected groups has to coincide with the actual formed group at a specific timestamp. Furthermore, the orientation of the participants to each other has heavy influence on the resulting RSSI values and therefore a normalization based on only proximity cannot be done.

Smoking is a leading risk factor negatively impacting the health of people, not only those partaking in it first-hand, but also to those around them. Different methods are available to assist people with quitting smoking, with various degrees of effectiveness. Researchers developing smoking cessation approaches would like to have controlled environments to test how effective they are before offering them as viable options. Virtual reality has been demonstrated to be an efficacious tool for facilitating the presentation of cues aimed at eliciting smoking cravings in the lab. Addiction, however, is a complex matter involving different parts of the brain, and how or when conditioned responses causing smoking cravings are activated depend on the individual in question. There are strong indications that personalization of smoking cues, or in other words using elements relevant in the addiction model of an individual, may elicit higher or at least consistent smoking cravings. The general aim is that developing smoking cessation approaches targeting those most relevant elements may have higher ecological validity and therefore be more effective in assisting people both with quitting smoking and maintaining that behavior change. To assist researchers with testing their smoking cessation approaches using a virtual coach, we have created a system enabling the presentation of virtual environments and facilitating communication between researchers and participants while the latter are viewing them. We have evaluated the effectiveness of our system in eliciting a familiar experience, which we posit is a major component tied to smoking cravings. Our results showed that personalized virtual environments elicited a more familiar experience than non-personalized ones. We also examined the usability of our developed user interface, as well as the sense of presence elicited by our system, both of which received positive scores.

Recent advancements in quantification of repair outcomes of CRISPR-Cas9 mediated double-stranded DNA breaks (DSBs) have allowed for the use of machine learning for predicting the frequencies of these repair outcomes. Local DNA sequence context influences the frequencies of mutations that arise when DNA gets repaired after it is targeted by CRISPR (CRISPR outcomes). Contemporary models exploit this and can predict what the frequencies are of CRISPR outcomes at predetermined genomic loci. Predictions of such models are reasonably precise, but there may be opportunities for improvement in how the DNA sequence context is leveraged for making predictions. Some models only utilize a set of hand-crafted features, limiting the available information for the model. Other models do utilize broader sequence context but disregard sequence order or only predict a limited set of outcome classes. In this work we present an attention-based deep learning model that uses DNA sequence context to make fine-grained CRISPR outcome predictions. We present a custom input embedding for representing DSB repair outcomes and we expand on existing methods for analyzing attention-based models.

This thesis has researched how the Midge compares to a modern mobile phone regarding the accuracy and reliability of the rotation vector from the DMP in both devices. The rotation of the main axis of the Midge accurately matches that of the modern mobile phone, which means that the accuracy and reliability of the rotation vector from the DMP of both devices are mostly similar. The rotation of the secondary axes of the Midges do not exactly match that of the mobile phone. Additionally, the rotation of the secondary axes differs for each Midge. These differences might suggest that the DMP of the Midges are less accurate at detecting small changes. Further research is required to draw a definite conclusion.

When analysing social interactions, manual labour is often required to identify what is happening. An automated method of detecting who is interacting with who would already prove to be a significant help. This paper looks at how automated interaction detecting can be established. We look at methods of detecting proximity and look deeper into detecting F-formations using proximity. An f-formation is a group of people who are standing together with the intention of conversing. We show that it is possible to detect f-formations using a data-set containing proximity information and f-formations as ground truths. Our results show that using only proximity data from this dataset; we can detect f-formations better than the baseline provided in that dataset.

The Midge is a sensor device developed by the Socially Perceptive Computing Lab (SPCL) at Delft University of Technology (TU Delft). This device is used to monitor human behaviour in social settings using several sensors. In this paper, the accuracy of the Inertial Measurement Unit (IMU) chip used in the Midge sensor package was evaluated. The IMU is responsible for sensing motion in multiple directions using an accelerometer, gyroscope and magnetometer. This experiment was performed by comparing several Midge devices with each other as well as comparing them with a modern smartphone. First will be explained how the control software was updated to run on modern hardware (computers) and the work that was done to reliably convert the generated binary data to readable data (parsing). Then the process of creating the test setup, performing the tests and analyzing the data will be explained.

The goal for this paper is to find out what the smart badge provided by the Social Perceptive Computive Lab (SPCL) group is and what it contains. The sensors that are used in the smart badge are the Accelerometer, Gyroscope and Magnetometer. The main question of this paper is ”What is the benefit of using full 9-DOF IMU data in predicting speaking status, as opposed to using only accelerometer signals?”. The three senors all contribute in their own way and complement each other to give an estimate about the speaking status. The ability to estimate the speaking status using the smart badge opens up the potential for analyzing more about the social aspects of people without the need to record what they are saying.