Markerless 3d face tracking

Christian Walder, Martin Breidt, Heinrich Bulthoff, Bernhard Schölkopf, Cristóbal Curio

Research output: Chapter in Book/Report/Conference proceedingConference contribution

9 Citations (Scopus)

Abstract

We present a novel algorithm for the markerless tracking of deforming surfaces such as faces. We acquire a sequence of 3D scans along with color images at 40Hz. The data is then represented by implicit surface and color functions, using a novel partition-of-unity type method of efficiently combining local regressors using nearest neighbor searches. Both these functions act on the 4D space of 3D plus time, and use temporal information to handle the noise in individual scans. After interactive registration of a template mesh to the first frame, it is then automatically deformed to track the scanned surface, using the variation of both shape and color as features in a dynamic energy minimization problem. Our prototype system yields high-quality animated 3D models in correspondence, at a rate of approximately twenty seconds per timestep. Tracking results for faces and other objects are presented.

Original languageEnglish
Title of host publicationLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Pages41-50
Number of pages10
Volume5748 LNCS
DOIs
Publication statusPublished - 2009 Nov 4
Externally publishedYes
Event31st Annual Symposium of the Deutsche Arbeitsgemeinschaft fur Mustererkennung, DAGM 2009 - Jena, Germany
Duration: 2009 Sep 92009 Sep 11

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume5748 LNCS
ISSN (Print)03029743
ISSN (Electronic)16113349

Other

Other31st Annual Symposium of the Deutsche Arbeitsgemeinschaft fur Mustererkennung, DAGM 2009
CountryGermany
CityJena
Period09/9/909/9/11

Fingerprint

Face Tracking
Color
Implicit Surfaces
Nearest Neighbor Search
Partition of Unity
Energy Minimization
Color Image
3D Model
Minimization Problem
Registration
Template
Correspondence
Mesh
Face
Prototype
Object

ASJC Scopus subject areas

  • Computer Science(all)
  • Theoretical Computer Science

Cite this

Walder, C., Breidt, M., Bulthoff, H., Schölkopf, B., & Curio, C. (2009). Markerless 3d face tracking. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) (Vol. 5748 LNCS, pp. 41-50). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 5748 LNCS). https://doi.org/10.1007/978-3-642-03798-6_5

Markerless 3d face tracking. / Walder, Christian; Breidt, Martin; Bulthoff, Heinrich; Schölkopf, Bernhard; Curio, Cristóbal.

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Vol. 5748 LNCS 2009. p. 41-50 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 5748 LNCS).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Walder, C, Breidt, M, Bulthoff, H, Schölkopf, B & Curio, C 2009, Markerless 3d face tracking. in Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). vol. 5748 LNCS, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 5748 LNCS, pp. 41-50, 31st Annual Symposium of the Deutsche Arbeitsgemeinschaft fur Mustererkennung, DAGM 2009, Jena, Germany, 09/9/9. https://doi.org/10.1007/978-3-642-03798-6_5
Walder C, Breidt M, Bulthoff H, Schölkopf B, Curio C. Markerless 3d face tracking. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Vol. 5748 LNCS. 2009. p. 41-50. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). https://doi.org/10.1007/978-3-642-03798-6_5
Walder, Christian ; Breidt, Martin ; Bulthoff, Heinrich ; Schölkopf, Bernhard ; Curio, Cristóbal. / Markerless 3d face tracking. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Vol. 5748 LNCS 2009. pp. 41-50 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).
@inproceedings{977ddbf7ca2844f78e159de0edfc0aed,
title = "Markerless 3d face tracking",
abstract = "We present a novel algorithm for the markerless tracking of deforming surfaces such as faces. We acquire a sequence of 3D scans along with color images at 40Hz. The data is then represented by implicit surface and color functions, using a novel partition-of-unity type method of efficiently combining local regressors using nearest neighbor searches. Both these functions act on the 4D space of 3D plus time, and use temporal information to handle the noise in individual scans. After interactive registration of a template mesh to the first frame, it is then automatically deformed to track the scanned surface, using the variation of both shape and color as features in a dynamic energy minimization problem. Our prototype system yields high-quality animated 3D models in correspondence, at a rate of approximately twenty seconds per timestep. Tracking results for faces and other objects are presented.",
author = "Christian Walder and Martin Breidt and Heinrich Bulthoff and Bernhard Sch{\"o}lkopf and Crist{\'o}bal Curio",
year = "2009",
month = "11",
day = "4",
doi = "10.1007/978-3-642-03798-6_5",
language = "English",
isbn = "3642037976",
volume = "5748 LNCS",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
pages = "41--50",
booktitle = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

}

TY - GEN

T1 - Markerless 3d face tracking

AU - Walder, Christian

AU - Breidt, Martin

AU - Bulthoff, Heinrich

AU - Schölkopf, Bernhard

AU - Curio, Cristóbal

PY - 2009/11/4

Y1 - 2009/11/4

N2 - We present a novel algorithm for the markerless tracking of deforming surfaces such as faces. We acquire a sequence of 3D scans along with color images at 40Hz. The data is then represented by implicit surface and color functions, using a novel partition-of-unity type method of efficiently combining local regressors using nearest neighbor searches. Both these functions act on the 4D space of 3D plus time, and use temporal information to handle the noise in individual scans. After interactive registration of a template mesh to the first frame, it is then automatically deformed to track the scanned surface, using the variation of both shape and color as features in a dynamic energy minimization problem. Our prototype system yields high-quality animated 3D models in correspondence, at a rate of approximately twenty seconds per timestep. Tracking results for faces and other objects are presented.

AB - We present a novel algorithm for the markerless tracking of deforming surfaces such as faces. We acquire a sequence of 3D scans along with color images at 40Hz. The data is then represented by implicit surface and color functions, using a novel partition-of-unity type method of efficiently combining local regressors using nearest neighbor searches. Both these functions act on the 4D space of 3D plus time, and use temporal information to handle the noise in individual scans. After interactive registration of a template mesh to the first frame, it is then automatically deformed to track the scanned surface, using the variation of both shape and color as features in a dynamic energy minimization problem. Our prototype system yields high-quality animated 3D models in correspondence, at a rate of approximately twenty seconds per timestep. Tracking results for faces and other objects are presented.

UR - http://www.scopus.com/inward/record.url?scp=70350437838&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=70350437838&partnerID=8YFLogxK

U2 - 10.1007/978-3-642-03798-6_5

DO - 10.1007/978-3-642-03798-6_5

M3 - Conference contribution

AN - SCOPUS:70350437838

SN - 3642037976

SN - 9783642037979

VL - 5748 LNCS

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 41

EP - 50

BT - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

ER -