This research explores the integration of episodic memory and visual short-term memory in social robots to enhance engagement and interaction with older adults, particularly those with dementia, through art activities such as painting. Dementia significantly impacts millions globally, being one of the leading causes of death worldwide. Furthermore, it creates a substantial burden on caregivers and healthcare systems. This research aims to leverage social robots to provide meaningful engagement, companionship, and emotional support to individuals with dementia.

The study involved developing a painting application using Android Studio and a conversational agent using generative AI. The AI model was enhanced with episodic memory and visual short-term memory, enabling it to reference past interactions through episodic memory and analyze painting progress through recent visual inputs with short-term memory. This qualitative study included 8 participants, and post-session interviews were conducted to gain deeper insights into the participants' experiences with the system.

The study suggests that memory-enhanced social robots can improve engagement and interaction quality among older adults interested in painting, but have limited impact on those uninterested in the activity. Mixed responses were observed regarding the robot's role as a companion, with some participants feeling a sense of company while others did not. Overall, participants did not experience frustration due to the robot's presence.

Challenges include response time and real-time awareness, highlighting the need for a more intelligent system. Given the small sample size of eight participants, future research should involve larger groups for more comprehensive and reliable data.

Learning a Foreign Language is crucial in today’s interconnected world, as it enhances cross-cultural communication and fosters personal growth. Despite its many benefits, learners often face significant challenges, primarily due to Anxiety. To address these challenges, we introduce ELLA, a Dutch tutoring system powered by GPT-4. ELLA features an innovative Episodic Memory architecture, enabling learners to review and delete system-generated episodes from past interactions, thereby giving them a sense of control. Through this thesis, we explore the impact of Episodic Memory on Foreign Language Learners' Speaking Anxiety and the associated learning benefits. The investigation is divided into four sub-questions: the relationship between Foreign Language Anxiety and Language Learning, the optimization of the Episodic Memory framework for Learning, the influence of Episodic Memory on speaking-related State Anxiety, and the perceived benefits for learners with Foreign Language Anxiety. Our experimental study demonstrated that ELLA has the potential to reduce speaking-related State Anxiety in Dutch learners and offers significant learning benefits as perceived by users. However, areas for improvement are identified, such as the need for an Automatic Speech Recognition system that accurately recognizes foreign accents and a Large Language Model fine-tuned for the Language Learning task. Future research should focus on these enhancements, evaluate the system in different languages using standardized pre-and post-tests, and involve longer interactions with a larger and more diverse user base over time.

Recognizing facial emotions is key for social interaction, yet the subjective nature of emotion labeling poses challenges for automatic facial affect prediction. Variability in how individuals interpret emotions leads to uncertainty in training data for machine learning models. While multiple raters and interrater agreement (IRA) measures are used to address this, the extent of their use and their impact on dataset reliability is not well understood. This systematic literature review investigates the methodologies used to measure IRA in facial affect recognition datasets. Concrete eligibility and feasibility criteria were applied, and it resulted in 47 papers being retrieved from Scopus, Web of Science, IEEExplore, and ACM Digital Library. Data on affect states, affect representation schemes (ARS), and IRA methodologies used by the datasets and their corresponding papers were extracted to provide a comprehensive overview and allow a detailed analysis. Clear correlation was not found in between ARS and IRA, but the retrieved data showed that Fleiss' kappa was the most popular methodology over time but also in the recent years.

With the advancement of technology and the integration of Large Language Models, interactive robots are entrusted with more significant tasks that contribute to human well-being. This transition is particularly significant in educational settings, where social robots assume diverse roles such as teacher, tutor, and instructional tool, aiming to enhance learning experiences. Engagement emerges as a central theme in Human-Robot Interaction studies, especially within pedagogical contexts, where it has a significant impact on learning outcomes. However, optimizing engagement rather than maximizing it may yield superior learning results. This thesis investigates the correlation between the employment of engagement strategies and learning outcomes in prompt-driven discussions with robots, focusing on facilitating self-reflection and understanding personal values among children. Due to limitations with the target population, a total of 55 university students conversed with the robot in two different sessions, where they discussed different situations, how they would react in each, and the reasoning behind their behavior, using the Schwartz values as a basis for the choices. The participants were divided into two conditions, to examine the effect of techniques such as motivational interviewing, cues to images, feedback, and the use of a memory model on the quality of their arguments and their ability to recollect the interaction. Results suggest that the strategies lead to significantly more reflective arguments for the first session, but they equalize in the second interaction. No significant effect was found between condition and participants' identifying their value profile which led to an examination of the assumptions made by the robot. Participants who agreed with the assumptions were more likely to identify their values correctly, but, generally, the value model was unable to fully capture the intricacies of human motivation. Participants found the robot equally likable, a possible result of the novelty effect. On recollection quality, the data shows that the effect of the session is too strong to allow room for the condition. Overall, this study contributes valuable insights into the influence of engagement strategies on learning outcomes in educational interactions, offering guidance for designing more effective and engaging interactions in the future.

The promotion of desirable behaviours, such as socially appropriate or health-promoting actions, can be bolstered through a deeper understanding and awareness of the values that underpin the associated behavioural choices. Various implementations for promoting behaviour change based on goals already exist in Human-Robot Interaction, but, since values are the building blocks for our behaviour an agent can use values for behaviour change. By reflecting on value-related choices in a conversation, a conversational agent can assist in identifying the values at stake. This is particularly significant as these agents serve as accessible and non-judgemental platforms for discussing various concerns in a private setting. Such reflective conversations are time-consuming and may span multiple sessions, necessitating some form of memory (e.g., to reflect on earlier statements and compare the choices made for different situations).

While various robotic agents have been developed to provide behavioural support (e.g., for human health), the absence of a comprehensive memory structure and dialogue strategies capable of fostering personalised, reflective conversations based on the appreciation of certain values and actions in various scenarios through contextualised values remains a challenge. To address this, this study introduces the Contextualised Value Model – a dynamic memory model designed to facilitate value-based reflection and support personalised interactions between humans and robotic agents.

To realise this robotic memory, a conversational agent was designed that could elicit values from participants by discussing various scenarios that happen in daily life and reflecting on said values using perspective-taking and other dialogue strategies.

The evaluation of the Contextualised Value Model focused on three primary aspects: the model's accuracy, the influence on likeability and intelligence, and the effect on participants' value awareness. The model was evaluated during a between-subjects experiment (N=54), consisting of two conditions, one where the robot was able to update and use the Contextualised Value Model, and another one where the Contextualised Value Model was random throughout the conversation.

The outcome measures indicated that the integration of the memory model in conversations led to a personalised and relevant conversation, highlighting the potential of the Contextualised Value Model in enhancing conversation personalisation. Although participants' value awareness and perception of the robot's likeability and intelligence did not significantly differ based on the memory model, the study emphasised the need for extended observation to thoroughly evaluate long-term impacts.

Overall, the Contextualised Value Model presents a promising framework for enhancing personalised interactions in various real-world applications, emphasising the need for further research in this area. The ePartner4all project could be further developed to complement the efforts of primary school teachers and parents in supporting children's self-learning of socially, mentally, and physically desirable behaviours.

Robots in classroom settings can help teachers with providing personalised attention to children's health and development. As part of this personalisation, robots should store and use (verbal or multi-modal) information about the children they interact with. One aspect that has been unexplored in existing literature is the detection of memorable moments during these child-robot interaction. Eye-gaze tracking is a low cost and non-invasive method applied widely to gain insight into human's inner processes. This study has found that several state-of-the-art time series machine learning models perform better-than-chance on the detection of memorable moments using gaze tracking.
In addition, a shapelet-based transform classifier also performed better-than-chance in distinguishing memories according to $3$ different levels of recall detail. Manual data analysis has identified significantly different gaze behaviour during memorable moments and not memorable moments as well as in the gaze behaviour for different levels of recall detail. The comparison of the results with related literature leads to the hypothesis that memorable moments are likely to be moments of both high levels of engagement and deep thinking. The data analyses also provided insight into children's gaze behaviour for different reasons for remembering a moment. The results show that these reasons, or `internal processes', are distinguishable by gaze patterns and thus provide insight into items or concept that draw the child's attention. This study shows that memorable moments detection for children is a developing and promising field that could potentially provide a lot of insight into children's situated thought processes.

Advances in artificial intelligence and machine learning have led to a steep rise in the adoption of AI to augment or support human decision-making across domains.
There has been an increasing body of work addressing the benefits of model interpretability and explanations to help end-users or other stakeholders decipher the inner workings of the so-called "black box AI systems".
Yet, little is currently understood about the role of modalities through which explanations can be communicated (e.g. text, visualizations, or audio) to inform, augment, and shape human decision-making.
This thesis addresses this research gap through the lens of a credibility assessment system.
Considering the deluge of information available through various channels, people constantly make decisions while considering the perceived credibility of the information they consume.
However, with an increasing information overload, assessing the credibility of the information we encounter is a non-trivial task.
To help users in this task, automated credibility assessment systems have been devised as decision support systems in various contexts (e.g. assessing the credibility of news or social media posts).
However, for these systems to be effective in supporting users, they need to be trusted and understood.
Explanations have been shown to play an essential role in informing users' reliance on decision support systems.
This thesis investigates the influence of explanation modalities on an AI-assisted credibility assessment task.
A between-subjects experiment (N=375) was performed, spanning six different explanation modalities, to evaluate the role of explanation modality on the accuracy of AI-assisted decision outcomes, the perceived system trust among users, and system usability.
The results indicate that explanations play a significant role in shaping users' reliance on the decision support system and, thereby, the accuracy of decisions made.
Users were found to perform with higher accuracy while assessing the credibility of statements in the presence of explanations.
Users additionally had a significantly harder time agreeing on statement credibility without explanations. With explanations present, text and audio explanations were more effective than graphic explanations.
This thesis concludes that combinations of graphical and text and/or audio explanations were significantly effective. Such combinations of modalities led to a higher user performance than using graphical explanations alone.

Negotiation is not a skill that comes naturally to most people. However, most people could benefit from attaining good negotiation skills. Non-verbal behaviour plays an important role in negotiations. Previous studies have shown a link between mimicry through conversational agents and self-regulation of non-verbal behaviour. The current study aims to raise a person’s self-awareness of their non-verbal behaviour through the medium of a conversational agent in negotiation training. An experiment (N = 64) is conducted where participants performed negotiations with an intermediate feedback round where the participants' non-verbal behaviour was played back on a virtual agent. The agent either mimicked the participants’ non-verbal behaviour or exhibited idle behaviour. It was found that the mimicking agent provided better feedback to the participants. Furthermore, repeated negotiations increase a person's self-awareness. The data indicate that raising a person’s self-awareness in negotiation training by utilising non-verbal mimicry through a conversational agent is feasible.

In this work, a conversational interaction was designed and implemented to test the effect of references to past events or shared experiences rephrased into motivational phrases within the context of working towards a diet related goal that can assist with type II diabetes over multiple sessions.
Prior works that utilized a memory typically did not utilize a memory to refer to the past within conversations and when past events were referred to within a conversation, they were not utilized with motivational rephrasing. Most prior works did not analyze long term interaction or how references to past events should have been utilized. To further research in this area, the following research questions were posed.
1.) Does referring to previous sessions improve goal attainment?
2.) Does referring to previous sessions improve user experience?
3.) Does a variety of references differ from making the same reference?
To determine the answers to these questions, interactions were carried out through Zoom in a between subjects experiment with three groups of participants.
The results, once analyzed, found that there was a significant difference between interactions that only make references to the same shared experience and interactions that make references to various shared experiences where various shared experiences resulted in a significant increase in the change in motivation from prior to the experiment to during the experiment.
This offers the following contributions:
1.) Provide clarity on how shared experiences should be used and not only whether they should be used.
2.) A motivational memory utilizing shared experiences with intrinsic motivational value
3.) Determine the effect of motivational references on goal achievement and user experience.
This holds promise for the use of such a memory for goal based efforts and future work can further the domains of application and effectiveness of the motivational memory and shared experiences.