HandyTool: Object Manipulation Through Metaphorical Hand/Fingers-to-Tool Mapping

Eunbi Seol, Gerard J. Kim

Research output: Chapter in Book/Report/Conference proceedingConference contribution

2 Citations (Scopus)


In this paper, we introduce “HandyTool” a method (and an interface) for virtual object manipulation based on a metaphorical/structural mapping of various everyday tools to our hands and fingers. The basic idea is to virtually transform the hand/fingers into a proper tool (e.g. a fist becoming a hammer head) and gesturally apply it (e.g. hammer in to insert) to and manipulate the target object (e.g. a nail) directly. The main intended objective of HandyTool is to enhance the tool usage experience by one (or one’s body part) becoming the tool itself and thereby also possibly improving the task performance. A usability experiment was carried out to assess the projected merits, comparing HandyTool to the case of the as-is emulation of the tool usage (i.e. the tracked hand/finger controlling the tool to apply it to the target object) and to the case of using the controller. Our experiment was not able to show the clear and full potential of HandyTool because of the current performance limitation of the hand/fingers tracking sensor and due to the simplicity in the structural mapping between the tool and hand/fingers. The structural metaphor itself was still shown to be helpful when the controller was used (i.e. stable sensing).

Original languageEnglish
Title of host publicationHCI International 2019 - Posters - 21st International Conference, HCII 2019, Proceedings
EditorsConstantine Stephanidis
PublisherSpringer Verlag
Number of pages8
ISBN (Print)9783030235277
Publication statusPublished - 2019
Event21st International Conference on Human-Computer Interaction, HCI International 2019 - Orlando, United States
Duration: 2019 Jul 262019 Jul 31

Publication series

NameCommunications in Computer and Information Science
ISSN (Print)1865-0929
ISSN (Electronic)1865-0937


Conference21st International Conference on Human-Computer Interaction, HCI International 2019
Country/TerritoryUnited States


  • Interactive learning environments
  • Mixed/augmented reality
  • Virtual reality

ASJC Scopus subject areas

  • Computer Science(all)
  • Mathematics(all)


Dive into the research topics of 'HandyTool: Object Manipulation Through Metaphorical Hand/Fingers-to-Tool Mapping'. Together they form a unique fingerprint.

Cite this