Using affective computing to enhance the believability of virtual agents and the social interaction with users

Abstract

The drive to create believable virtual agents capable of engaging in natural social interactions with humans has fuelled extensive research efforts in affective computing, the field dedicated to developing systems that recognize and adapt to human emotions. Despite advancements in recent decades, the field continues to face challenges related to (1) responding accurately to human emotions and fostering a continuous affective loop, (2) finding reliable stimuli to elicit a wide range of affective states, (3) processing affective cues appropriately and (4) generalizing affect recognition across contexts. In an attempt to address these issues, this work proposed identifying general affective states that include thematically similar emotions. Equipping agents with a model capable of detecting such sets would enable them to provide thematically close responses, enhancing their emotional coherence.

To assess the potential of the approach, the first phase of the project used the eight subspaces of the Pleasure-Arousal-Dominance (PAD) model to define the general affective states of the user and to design agent responses. An experiment was conducted and included interactions with two agents that dynamically adjusted facial expressions and text in response to the eight general states. The second phase of the project explored horror games as a medium to elicit anxiety-related emotions within the ”Anxious” PAD subspace. A new tool was designed to help collect affect annotation over time. These annotations, along with physiological measures generated a new dataset. Using this dataset, classifiers were built to identify subclasses within the P, A and D components of the selected subspace, while investigating how varying the size and offset of time windows for processing physiological signals affected the classifiers' performance. The final phase of the project explored routes for context generalization using the proposed adaptive windowing technique.

Results indicated a positive overall perception of the agent’s believability when responding to eight general affective states, with further improvement observed when changes in the user’s physiological arousal were included. Adjusting the window settings for processing physiological signals led to improved prediction performance. Moreover, the implementation of the adaptive windowing technique demonstrated the potential to improve the model's ability to better characterize affective states across diverse datasets.

Date of Award	25 Feb 2025
Original language	English
Awarding Institution	Abertay University
Sponsors	Northwood Charitable Trust
Supervisor	George Lovell (Supervisor) & Kenneth Fee (Supervisor)