This project aims to harness the potential of music-making and robotic interaction to enhance creative expression and cognitive function among individuals with cognitive impairment and dementia. With the aging population, there is a growing demand for innovative interventions that support cognitive health and active engagement. Music therapy has demonstrated effectiveness in stimulating cognitive function and emotional expression in individuals with dementia. Creative expression through music serves as a unique outlet, fostering cognitive functions and emotional well-being. This study explores the synergy between music therapy and Socially Assistive Robots (SARs) to develop a more immersive therapeutic intervention. It represents an exploratory investigation into an end-to-end robotic intervention, proposing various interaction elements and examining their functionalities. Each element is designed to foster engagement, enhance perceptions of collaboration, and promote feelings of creativity.
In this thesis, we propose an end-to-end interactive music-making experience designed for use with the Pepper robot, an SAR. The system features a user-friendly interface with eight color-coded boxes, each corresponding to a musical note. Users simply tap the boxes to create melodies. The Pepper robot acts as a guide, assisting users in interacting with the interface. It additionally implements an engagement tracking system by monitoring user interaction through the screen taps on the interface and provides real-time feedback and encouragement. If a period of inactivity is detected, Pepper gently nudges the user to re-engage. Furthermore, the robot functions as a collaborative musical partner, providing rhythmic accompaniment if the user desires. The system also records user-created music and provides playback functionality, allowing users to revisit their compositions.
Methodologically, the study involves an end-to-end system comprising an intelligent music-making interface and an interactive robot providing real-time feedback and rhythmic accompaniment. Insights from the exploratory study highlight the benefits of real-time feedback in enhancing engagement, particularly among participants with musical backgrounds. However, rhythmic accompaniment shows mixed results in fostering collaboration, indicating a preference among participants for emotional connection in collaborative settings. Since this is an exploratory study, the empirical study focuses on healthy older adults, a population with an increased risk of cognitive decline. This is because individuals with dementia are a vulnerable group. Music interventions have shown promise in improving cognitive function and engagement in individuals with dementia. Therefore, this study informs the design of future interventions for people with early-stage dementia.\\
Key findings underscore the potential of real-time feedback and interaction in promoting engagement in the activity. The intelligent music interface also shows potential to support creative exploration, albeit with improvements needed for advanced musical participants. Participants appreciate the playback feature, enhancing their sense of creative ownership and motivation. Despite promising outcomes, the study acknowledges limitations in sample size and participant demographics, primarily recruiting from music-engaged older adults rather than the target demographic of individuals with cognitive impairments.
Future research directions include expanding participant diversity, refining robot interaction capabilities, and addressing technical challenges to improve system usability and accessibility. Integrating findings from ongoing research on music and memory could further enhance personalized interventions. Ultimately, this study lays the groundwork for future developments in robotic interventions that promote well-being through music therapy for individuals with cognitive impairments.

Cognitive Impairment, commonly termed as Dementia, affects a large number of older adults. People with dementia (PwD) experience cognitive decline that impacts their ability to perform daily activities and maintain social connections. The number of PwD is expected to rise, and unfortunately, there is still no cure. A number of non drug therapies such as music and art therapy are used to stimulate neurons and slow down the progress of the disease. When symptoms become severe, full-time care may be needed, requiring a move to a nursing home for formal care. However, studies show that interaction with peers is limited in a care home, leading to loneliness and reduced quality of life. Caregivers organize group activities like singing and baking to alleviate these feelings, but their busy schedule can limit the frequency of these exercises. This begs the question: Can a robot support caregivers in the provision of group activities in a care home? This would allow caregivers more time to attend to the personal needs of the residents. It not only has the potential to reduce workload of caregivers but also creates opportunities for residents to interact with each other.

In this thesis, we propose the design of a robot which moderates a sing-along activity for a group of people living in a nursing home. An activity in the realm of music therapy is chosen since it is one of the non-drug therapies which has shown to have numerous benefits like improving memory recall, eliciting emotions and aiding relaxation. The robot session is curated to give opportunities for interaction with group members and with the robot too. While the song is playing and participants are singing along with the music, the robot performs engagement detection, and encourages those who do not seem involved.

Multiple ways are proposed to detect engagement in HRI, such as emotion recognition and estimation of head pose. These approaches have limitations in the scope of this thesis, particularly because these methods are not designed for groups of older adults residing in a care home. Given these limitations and our focus on the target group, we propose a novel engagement detection methodology, which leverages the presence of a robot and assigns an active role to it. Instead of relying on conventional passive engagement detection methods, this approach uses an interactive probing technique. The robot prompts users to perform certain actions, and whether or not they respond to it is monitored using pose estimation. This is termed robot engagement. This approach is combined with activity engagement, which indicates whether the participant is singing or not. With this hybrid technique, we aim to increase the efficacy of engagement detection. If a participant is detected as disengaged, either in the activity or in the robot, we provide personalized encouragement and motivation by addressing participants by name. Along with this, the robot also compliments those who actively participate, motivating them further...

Several useful technologies to help people with dementia already exist. Robotic pets, for example, have been shown to alleviate negative symptoms associated with dementia. Most robotic pets, however, only have pet-like functionality. This is useful on its own, but adding additional care functionality would be helpful. A different useful tool for people with dementia is reminder systems. These systems can help them remember to do important daily tasks, like eating meals and taking medicine. These systems, however, usually use a phone as a medium. The user might leave their phone in another room, and if they do not take it to the location where the task is to be completed, they cannot check what the reminder was for.

This thesis aims to combine these two systems, and presents the design of an integrated robot reminder system for people with dementia who live at home. It aims to address the problem of people with dementia forgetting important things, improve their quality of life, and reduce the stress experienced by their informal caregivers. The system consists of an animal-like robotic pet, which delivers reminders for important activities of daily living directly to the user with dementia. It guides them to a location in the house, where a screen tells them what the reminder is for. Through sensors placed in the house, the system can automatically detect whether some reminders have been completed, and can use contextual clues to send reminders at the best time. The research question for this thesis is “How can an integrated robotic reminder system effectively support people with dementia living at home in completing daily tasks while alleviating stress of their informal caregiver?”

Three research topics are explored in the thesis. The first topic is that of value-sensitive design. The two main user groups, people with dementia and caregivers, both have values they want to see reflected in the system. Sometimes, these values may clash. The conflict between these values is explored in this thesis. Secondly, the interaction between the robot and the person in the context of a reminder system is researched. It should be clear to a user that the robot is trying to convey a reminder. The third topic relates to the software architecture and software engineering requirements. Private data should be stored securely, and the requirements should be written to make future development easy. Recommendations are made regarding the ideal setup of the system, ensuring proper security and usefulness. These technical choices were validated and improved through two interviews with experts in the field of privacy.

The system was designed through an iterative process. In order to improve the system and validate its acceptability, two design workshops were held, one with professional caregivers and one with people with dementia living in a care home. We showed them a high-fidelity robot prototype, and showed the caregivers a user interface prototype for a caregiver app. These workshops showed that the combination of the robot and the reminder functionality was appealing, though not everyone was as interested.

Joint attention is the shared focus multiple people can have on the same object and it is subconsciously used by humans every day. The simple act of verbally or non-verbally pointing out an object to one another, is a form of joint attention. Its use facilitates human cooperation, such as when someone needs to hand over an object to another person. When robots display those same joint attention behaviours, it could be useful in improving human-robot interactions and perhaps variably so in various levels of jointness.
This research investigates the effects of these levels of jointness on the interactions between a human and a robot. This is done by investigating the effect on the person's task performance, the effect on their mental model of the robot, and the effect on their perception of the robot as its own entity with its own mind. To this end, this research defines four levels of joint attention and designs an experiment that makes use of these four levels of joint attention. To perform this experiment, a system capable of establishing joint attention has been developed. The system is divided into two parts; hardware and software tools & applications which have not been developed in this project but are used as is, and software which has been actively developed in this research.
In this experiment, participants played three guessing games with a robot. Task performance was measured by the amount of time and hints needed, as well as the accuracy of a participant. The participant's mental model and perception of the robot as its own entity with its own mind were measured using a questionnaire.
This research found no significant effect on the mental model towards the robot and the perception of whether the robot is its own entity with its own mind, although it is not concluded that such an effect cannot be found.
This research did find a significant effect on task performance; higher levels of joint attention lead to faster task completion with less hints needed. But interestingly, this did not necessarily lead to a higher accuracy.
The system introduced by this research was originally intended to help children with autism spectrum disorder learn joint attention skills, but due to ethical and time constraints, the system was tested with and the experiment was performed with adults instead. Due to ethical constraints, the participants in this research were not asked to disclose whether they had an autism spectrum disorder diagnosis or not. However, future research could replicate this study with the intended target group.