Robust facial landmark extraction scheme using multiple convolutional neural networks

Hyungjoon Kim, Jisoo Park, Hyeon Woo Kim, Eenjun Hwang, Seungmin Rho

Research output: Contribution to journalArticlepeer-review

8 Citations (Scopus)


Facial landmarks are a set of features that can be distinguished on the human face with the naked eye. Typical facial landmarks include eyes, eyebrows, nose, and mouth. Landmarks play an important role in human-related image analysis. For example, they can be used to determine whether there is a human being in the image, identify who the person is, or recognize the orientation of a face when taking a photograph. General techniques for detecting facial landmarks can be classified into two groups: One is based on traditional image processing techniques, such as Haar cascade classifiers and edge detection. The other is based on machine learning techniques in which landmarks can be detected by training neural network using facial features. However, such techniques have shown low accuracy, especially in some special conditions such as low luminance and overlapped faces. To overcome these problems, we proposed in our previous work a facial landmark extraction scheme using deep learning and semantic segmentation, and demonstrated that with even a small dataset, our scheme could achieve reasonable facial landmark extraction performance under such conditions. Nevertheless, for more extensive dataset, we found several exceptional cases where the scheme could not detect face landmarks precisely. Hence, in this paper, we revise our facial landmark extraction scheme using a deep learning model called Faster R-CNN and show how our scheme can improve the overall performance by handling such exceptional cases appropriately. Also, we show how to expand the training dataset by using image filters and image operations such as rotation for more robust landmark detection.

Original languageEnglish
Pages (from-to)3221-3238
Number of pages18
JournalMultimedia Tools and Applications
Issue number3
Publication statusPublished - 2019 Feb 1


  • Convolutional neural networks
  • Facial landmark
  • Faster R-CNN
  • Object detection
  • Semantic segmentation

ASJC Scopus subject areas

  • Software
  • Media Technology
  • Hardware and Architecture
  • Computer Networks and Communications


Dive into the research topics of 'Robust facial landmark extraction scheme using multiple convolutional neural networks'. Together they form a unique fingerprint.

Cite this