Fusion-convbert: Parallel convolution and bert fusion for speech emotion recognition

Sanghyun Lee, David K. Han, Hanseok Ko

Research output: Contribution to journalArticlepeer-review

Abstract

Speech emotion recognition predicts the emotional state of a speaker based on the person’s speech. It brings an additional element for creating more natural human–computer interactions. Earlier studies on emotional recognition have been primarily based on handcrafted features and manual labels. With the advent of deep learning, there have been some efforts in applying the deep-network-based approach to the problem of emotion recognition. As deep learning automatically extracts salient features correlated to speaker emotion, it brings certain advantages over the handcrafted-feature-based methods. There are, however, some challenges in applying them to the emotion recognition problem, because data required for properly training deep networks are often lacking. Therefore, there is a need for a new deep-learning-based approach which can exploit available information from given speech signals to the maximum extent possible. Our proposed method, called “Fusion-ConvBERT”, is a parallel fusion model consisting of bidirectional encoder representations from transformers and convolutional neural networks. Extensive experiments were conducted on the proposed model using the EMO-DB and Interactive Emotional Dyadic Motion Capture Database emotion corpus, and it was shown that the proposed method outperformed state-of-the-art techniques in most of the test configurations.

Original languageEnglish
Article number6688
Pages (from-to)1-19
Number of pages19
JournalSensors (Switzerland)
Volume20
Issue number22
DOIs
Publication statusPublished - 2020 Nov 2

Keywords

  • Bidirectional encoder representations from transformers (BERT)
  • Convolutional neural networks (CNNs)
  • Fusion model
  • Representation
  • Spatiotemporal representation
  • Speech emotion recognition
  • Transformer

ASJC Scopus subject areas

  • Analytical Chemistry
  • Biochemistry
  • Atomic and Molecular Physics, and Optics
  • Instrumentation
  • Electrical and Electronic Engineering

Fingerprint Dive into the research topics of 'Fusion-convbert: Parallel convolution and bert fusion for speech emotion recognition'. Together they form a unique fingerprint.

Cite this