Multimodal similarity and categorization of novel, three-dimensional objects

Theresa Cooke, Frank Jäkel, Christian Wallraven, Heinrich Bulthoff

Research output: Contribution to journalArticle

47 Citations (Scopus)

Abstract

Similarity has been proposed as a fundamental principle underlying mental object representations and capable of supporting cognitive-level tasks such as categorization. However, much of the research has considered connections between similarity and categorization for tasks performed using a single perceptual modality. Considering similarity and categorization within a multimodal context opens up a number of important questions: Are the similarities between objects the same when they are perceived using different modalities or using more than one modality at a time? Is similarity still able to explain categorization performance when objects are experienced multimodally? In this study, we addressed these questions by having subjects explore novel, 3D objects which varied parametrically in shape and texture using vision alone, touch alone, or touch and vision together. Subjects then performed a pair-wise similarity rating task and a free sorting categorization task. Multidimensional scaling (MDS) analysis of similarity data revealed that a single underlying perceptual map whose dimensions corresponded to shape and texture could explain visual, haptic, and bimodal similarity ratings. However, the relative dimension weights varied according to modality: shape dominated texture when objects were seen, whereas shape and texture were roughly equally important in the haptic and bimodal conditions. Some evidence was found for a multimodal connection between similarity and categorization: the probability of category membership increased with similarity while the probability of a category boundary being placed between two stimuli decreased with similarity. In addition, dimension weights varied according to modality in the same way for both tasks. The study also demonstrates the usefulness of 3D printing technology and MDS techniques in the study of visuohaptic object processing.

Original languageEnglish
Pages (from-to)484-495
Number of pages12
JournalNeuropsychologia
Volume45
Issue number3
DOIs
Publication statusPublished - 2007 Jan 1
Externally publishedYes

Fingerprint

Touch
Weights and Measures
Technology
Research
Three Dimensional Printing

Keywords

  • Categorization
  • Haptics
  • MDS
  • Multisensory
  • Similarity

ASJC Scopus subject areas

  • Behavioral Neuroscience
  • Neuropsychology and Physiological Psychology

Cite this

Multimodal similarity and categorization of novel, three-dimensional objects. / Cooke, Theresa; Jäkel, Frank; Wallraven, Christian; Bulthoff, Heinrich.

In: Neuropsychologia, Vol. 45, No. 3, 01.01.2007, p. 484-495.

Research output: Contribution to journalArticle

@article{81a562b41c634206971598c6dc8b85d5,
title = "Multimodal similarity and categorization of novel, three-dimensional objects",
abstract = "Similarity has been proposed as a fundamental principle underlying mental object representations and capable of supporting cognitive-level tasks such as categorization. However, much of the research has considered connections between similarity and categorization for tasks performed using a single perceptual modality. Considering similarity and categorization within a multimodal context opens up a number of important questions: Are the similarities between objects the same when they are perceived using different modalities or using more than one modality at a time? Is similarity still able to explain categorization performance when objects are experienced multimodally? In this study, we addressed these questions by having subjects explore novel, 3D objects which varied parametrically in shape and texture using vision alone, touch alone, or touch and vision together. Subjects then performed a pair-wise similarity rating task and a free sorting categorization task. Multidimensional scaling (MDS) analysis of similarity data revealed that a single underlying perceptual map whose dimensions corresponded to shape and texture could explain visual, haptic, and bimodal similarity ratings. However, the relative dimension weights varied according to modality: shape dominated texture when objects were seen, whereas shape and texture were roughly equally important in the haptic and bimodal conditions. Some evidence was found for a multimodal connection between similarity and categorization: the probability of category membership increased with similarity while the probability of a category boundary being placed between two stimuli decreased with similarity. In addition, dimension weights varied according to modality in the same way for both tasks. The study also demonstrates the usefulness of 3D printing technology and MDS techniques in the study of visuohaptic object processing.",
keywords = "Categorization, Haptics, MDS, Multisensory, Similarity",
author = "Theresa Cooke and Frank J{\"a}kel and Christian Wallraven and Heinrich Bulthoff",
year = "2007",
month = "1",
day = "1",
doi = "10.1016/j.neuropsychologia.2006.02.009",
language = "English",
volume = "45",
pages = "484--495",
journal = "Neuropsychologia",
issn = "0028-3932",
publisher = "Elsevier Limited",
number = "3",

}

TY - JOUR

T1 - Multimodal similarity and categorization of novel, three-dimensional objects

AU - Cooke, Theresa

AU - Jäkel, Frank

AU - Wallraven, Christian

AU - Bulthoff, Heinrich

PY - 2007/1/1

Y1 - 2007/1/1

N2 - Similarity has been proposed as a fundamental principle underlying mental object representations and capable of supporting cognitive-level tasks such as categorization. However, much of the research has considered connections between similarity and categorization for tasks performed using a single perceptual modality. Considering similarity and categorization within a multimodal context opens up a number of important questions: Are the similarities between objects the same when they are perceived using different modalities or using more than one modality at a time? Is similarity still able to explain categorization performance when objects are experienced multimodally? In this study, we addressed these questions by having subjects explore novel, 3D objects which varied parametrically in shape and texture using vision alone, touch alone, or touch and vision together. Subjects then performed a pair-wise similarity rating task and a free sorting categorization task. Multidimensional scaling (MDS) analysis of similarity data revealed that a single underlying perceptual map whose dimensions corresponded to shape and texture could explain visual, haptic, and bimodal similarity ratings. However, the relative dimension weights varied according to modality: shape dominated texture when objects were seen, whereas shape and texture were roughly equally important in the haptic and bimodal conditions. Some evidence was found for a multimodal connection between similarity and categorization: the probability of category membership increased with similarity while the probability of a category boundary being placed between two stimuli decreased with similarity. In addition, dimension weights varied according to modality in the same way for both tasks. The study also demonstrates the usefulness of 3D printing technology and MDS techniques in the study of visuohaptic object processing.

AB - Similarity has been proposed as a fundamental principle underlying mental object representations and capable of supporting cognitive-level tasks such as categorization. However, much of the research has considered connections between similarity and categorization for tasks performed using a single perceptual modality. Considering similarity and categorization within a multimodal context opens up a number of important questions: Are the similarities between objects the same when they are perceived using different modalities or using more than one modality at a time? Is similarity still able to explain categorization performance when objects are experienced multimodally? In this study, we addressed these questions by having subjects explore novel, 3D objects which varied parametrically in shape and texture using vision alone, touch alone, or touch and vision together. Subjects then performed a pair-wise similarity rating task and a free sorting categorization task. Multidimensional scaling (MDS) analysis of similarity data revealed that a single underlying perceptual map whose dimensions corresponded to shape and texture could explain visual, haptic, and bimodal similarity ratings. However, the relative dimension weights varied according to modality: shape dominated texture when objects were seen, whereas shape and texture were roughly equally important in the haptic and bimodal conditions. Some evidence was found for a multimodal connection between similarity and categorization: the probability of category membership increased with similarity while the probability of a category boundary being placed between two stimuli decreased with similarity. In addition, dimension weights varied according to modality in the same way for both tasks. The study also demonstrates the usefulness of 3D printing technology and MDS techniques in the study of visuohaptic object processing.

KW - Categorization

KW - Haptics

KW - MDS

KW - Multisensory

KW - Similarity

UR - http://www.scopus.com/inward/record.url?scp=33845439323&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=33845439323&partnerID=8YFLogxK

U2 - 10.1016/j.neuropsychologia.2006.02.009

DO - 10.1016/j.neuropsychologia.2006.02.009

M3 - Article

C2 - 16580027

AN - SCOPUS:33845439323

VL - 45

SP - 484

EP - 495

JO - Neuropsychologia

JF - Neuropsychologia

SN - 0028-3932

IS - 3

ER -