Sign language spotting with a threshold model based on conditional random fields

Hee Deok Yang, Stan Sclaroff, Seong Whan Lee

Research output: Contribution to journalArticle

143 Citations (Scopus)

Abstract

Sign language spotting is the task of detecting and recognizing signs in a signed utterance, in a set vocabulary. The difficulty of sign language spotting is that instances of signs vary in both motion and appearance. Moreover, signs appear within a continuous gesture stream, interspersed with transitional movements between signs in a vocabulary and nonsign patterns (which include out-of-vocabulary signs, epentheses, and other movements that do not correspond to signs). In this paper, a novel method for designing threshold models in a conditional random field (CRF) model is proposed which performs an adaptive threshold for distinguishing between signs in a vocabulary and nonsign patterns. A short-sign detector, a hand appearance-based sign verification method, and a subsign reasoning method are included to further improve sign language spotting accuracy. Experiments demonstrate that our system can spot signs from continuous data with an 87.0 percent spotting rate and can recognize signs from isolated data with a 93.5 percent recognition rate versus 73.5 percent and 85.4 percent, respectively, for CRFs without a threshold model, short-sign detection, subsign reasoning, and hand appearance-based sign verification. Our system can also achieve a 15.0 percent sign error rate (SER) from continuous data and a 6.4 percent SER from isolated data versus 76.2 percent and 14.5 percent, respectively, for conventional CRFs.

Original languageEnglish
Pages (from-to)1264-1277
Number of pages14
JournalIEEE Transactions on Pattern Analysis and Machine Intelligence
Volume31
Issue number7
DOIs
Publication statusPublished - 2009

Keywords

  • Conditional random field
  • Sign language recognition
  • Sign language spotting
  • Threshold model

ASJC Scopus subject areas

  • Software
  • Computer Vision and Pattern Recognition
  • Computational Theory and Mathematics
  • Artificial Intelligence
  • Applied Mathematics

Fingerprint Dive into the research topics of 'Sign language spotting with a threshold model based on conditional random fields'. Together they form a unique fingerprint.

  • Cite this