Label Quality in AffectNet: Results of Crowd-Based Re-annotation

Doo Yon Kim, Christian Wallraven

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

AffectNet is one of the most popular resources for facial expression recognition (FER) on relatively unconstrained in-the-wild images. Given that images were annotated by only one annotator with limited consistency checks on the data, however, label quality and consistency may be limited. Here, we take a similar approach to a study that re-labeled another, smaller dataset (FER2013) with crowd-based annotations, and report results from a re-labeling and re-annotation of a subset of difficult AffectNet faces with 13 people on both expression label, and valence and arousal ratings. Our results show that human labels overall have medium to good consistency, whereas human ratings especially for valence are in excellent agreement. Importantly, however, crowd-based labels are significantly shifting towards neutral and happy categories and crowd-based affective ratings form a consistent pattern different from the original ratings. ResNets fully trained on the original AffectNet dataset do not predict human voting patterns, but when weakly-trained do so much better, particularly for valence. Our results have important ramifications for label quality in affective computing.

Original languageEnglish
Title of host publicationPattern Recognition - 6th Asian Conference, ACPR 2021, Revised Selected Papers
EditorsChristian Wallraven, Qingshan Liu, Hajime Nagahara
PublisherSpringer Science and Business Media Deutschland GmbH
Pages518-531
Number of pages14
ISBN (Print)9783031024436
DOIs
Publication statusPublished - 2022
Event6th Asian Conference on Pattern Recognition, ACPR 2021 - Virtual, Online
Duration: 2021 Nov 92021 Nov 12

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume13189 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference6th Asian Conference on Pattern Recognition, ACPR 2021
CityVirtual, Online
Period21/11/921/11/12

Keywords

  • AffectNet
  • Affective computing
  • Crowd annotation
  • Facial expression recognition

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Computer Science(all)

Fingerprint

Dive into the research topics of 'Label Quality in AffectNet: Results of Crowd-Based Re-annotation'. Together they form a unique fingerprint.

Cite this