Validation of Text Data Preprocessing Using a Neural Network Model

Ho Sung Woo, Ja Mee Kim, Won Gyu Lee

Research output: Contribution to journalArticlepeer-review

Abstract

Many artificial intelligence studies focus on designing new neural network models or optimizing hyperparameters to improve model accuracy. To develop a reliable model, appropriate data are required, and data preprocessing is an essential part of acquiring the data. Although various studies regard data preprocessing as part of the data exploration process, those studies lack awareness about the need for separate technologies and solutions for preprocessing. Therefore, this study evaluated combinations of preprocessing types in a text-processing neural network model. Better performance was observed when two preprocessing types were used than when three or more preprocessing types were used for data purification. More specifically, using lemmatization and punctuation splitting together, lemmatization and lowering together, and lowering and punctuation splitting together showed positive effects on accuracy. This study is significant because the results allow better decisions to be made about the selection of the preprocessing types in various research fields, including neural network research.

Original languageEnglish
Article number1958149
JournalMathematical Problems in Engineering
Volume2020
DOIs
Publication statusPublished - 2020

ASJC Scopus subject areas

  • Mathematics(all)
  • Engineering(all)

Fingerprint Dive into the research topics of 'Validation of Text Data Preprocessing Using a Neural Network Model'. Together they form a unique fingerprint.

Cite this