Contextual modulation of affect: Comparing humans and deep neural networks

Soomin Shin, Doo Yon Kim, Christian Wallraven

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Citation (Scopus)

Abstract

When inferring emotions, humans rely on a number of cues, including not only facial expressions, body posture, but also expressor-external, contextual information. The goal of the present study was to compare the impact of such contextual information on emotion processing in humans and two deep neural network (DNN) models. We used results from a human experiment in which two types of pictures were rated for valence and arousal: the first type depicted people expressing an emotion in a social context including other people; the second was a context-reduced version in which all information except for the target expressor was blurred out. The resulting human ratings of valence and arousal were systematically decreased in the context-reduced version, highlighting the importance of context. We then compared human ratings with those of two DNN models (one trained on face images only, and the other trained also on contextual information). Analyses of both categorical and the valence/arousal ratings showed that although there were some superficial similarities, both models failed to capture human rating patterns both in context-rich and context-reduced conditions. Our study emphasizes the importance of a more holistic, multi-modal training regime with richer human data to build better emotion-understanding systems in the area of affective computing.

Original languageEnglish
Title of host publicationICMI 2022 - Companion Publication of the 2022 International Conference on Multimodal Interaction
PublisherAssociation for Computing Machinery
Pages127-133
Number of pages7
ISBN (Electronic)9781450393898
DOIs
Publication statusPublished - 2022 Nov 7
Event24th ACM International Conference on Multimodal Interaction, ICMI 2022 - Bangalore, India
Duration: 2022 Nov 72022 Nov 11

Publication series

NameACM International Conference Proceeding Series

Conference

Conference24th ACM International Conference on Multimodal Interaction, ICMI 2022
Country/TerritoryIndia
CityBangalore
Period22/11/722/11/11

Keywords

  • DNNs
  • affective computing
  • contextual information
  • emotion recognition
  • user study

ASJC Scopus subject areas

  • Human-Computer Interaction
  • Computer Networks and Communications
  • Computer Vision and Pattern Recognition
  • Software

Fingerprint

Dive into the research topics of 'Contextual modulation of affect: Comparing humans and deep neural networks'. Together they form a unique fingerprint.

Cite this