Learning Free-Form Deformation for 3D Face Reconstruction from In-The-Wild Images

Harim Jung, Myeong Seok Oh, Seong Whan Lee

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

The 3D Morphable Model (3DMM), which is a Principal Component Analysis (PCA) based statistical model that represents a 3D face using linear basis functions, has shown promising results for reconstructing 3D faces from single-view in-the-wild images. However, 3DMM has restricted representation power due to the limited number of 3D scans and global linear basis. To address the limitations of 3DMM, we propose a straightforward learning-based method that reconstructs a 3D face mesh through Free-Form Deformation (FFD) for the first time. FFD is a geometric modeling method that embeds a reference mesh within a parallelepiped grid and deforms the mesh by moving the sparse control points of the grid. As FFD is based on mathematically defined basis functions, it has no limitation in representation power. Thus, we can recover accurate 3D face meshes by estimating the appropriate deviation of control points as deformation parameters. Although both 3DMM and FFD are parametric models, deformation parameters of FFD are easier to interpret in terms of their effect on the final shape. This practical advantage of FFD allows the resulting mesh and control points to serve as a good starting point for 3D face modeling, in that ordinary users can fine-tune the mesh by using widely available 3D software tools. Experiments on multiple datasets demonstrate how our method successfully estimates the 3D face geometry and facial expressions from 2D face images, achieving comparable performance to the state-of-the-art methods.

Original languageEnglish
Title of host publication2021 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2021
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages2737-2742
Number of pages6
ISBN (Electronic)9781665442077
DOIs
Publication statusPublished - 2021
Event2021 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2021 - Melbourne, Australia
Duration: 2021 Oct 172021 Oct 20

Publication series

NameConference Proceedings - IEEE International Conference on Systems, Man and Cybernetics
ISSN (Print)1062-922X

Conference

Conference2021 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2021
Country/TerritoryAustralia
CityMelbourne
Period21/10/1721/10/20

Keywords

  • 3D face reconstruction
  • 3D morphable model
  • Free-form deformation

ASJC Scopus subject areas

  • Electrical and Electronic Engineering
  • Control and Systems Engineering
  • Human-Computer Interaction

Fingerprint

Dive into the research topics of 'Learning Free-Form Deformation for 3D Face Reconstruction from In-The-Wild Images'. Together they form a unique fingerprint.

Cite this