A Two-Level Recurrent Neural Network Language Model Based on the Continuous Bag-of-Words Model for Sentence Classification

Yo Han Lee, Dong W. Kim, Myo Taeg Lim

Research output: Contribution to journalArticle

Abstract

In this paper, a new two-level recurrent neural network language model (RNNLM) based on the continuous bag-of-words (CBOW) model for application to sentence classification is presented. The vector representations of words learned by a neural network language model have been shown to carry semantic sentiment and are useful in various natural language processing tasks. A disadvantage of CBOW is that it only considers the fixed length of a context because its basic structure is a neural network with a fixed length of input. In contrast, the RNNLM does not have a size limit for a context but only considers the previous context's words. Therefore, the advantage of RNNLM is complementary to the disadvantage of CBOW. Herein, our proposed model encodes many linguistic patterns and improves upon sentiment analysis and question classification benchmarks compared to previously reported methods.

Original languageEnglish
Article number1950002
JournalInternational Journal on Artificial Intelligence Tools
Volume28
Issue number1
DOIs
Publication statusPublished - 2019 Feb 1

Fingerprint

Recurrent neural networks
Neural networks
Linguistics
Semantics
Processing

Keywords

  • continuous bag-of-words
  • language model
  • Recurrent neural network
  • sentence classification

ASJC Scopus subject areas

  • Artificial Intelligence

Cite this

@article{20f21806850340e99caedafce978ac2c,
title = "A Two-Level Recurrent Neural Network Language Model Based on the Continuous Bag-of-Words Model for Sentence Classification",
abstract = "In this paper, a new two-level recurrent neural network language model (RNNLM) based on the continuous bag-of-words (CBOW) model for application to sentence classification is presented. The vector representations of words learned by a neural network language model have been shown to carry semantic sentiment and are useful in various natural language processing tasks. A disadvantage of CBOW is that it only considers the fixed length of a context because its basic structure is a neural network with a fixed length of input. In contrast, the RNNLM does not have a size limit for a context but only considers the previous context's words. Therefore, the advantage of RNNLM is complementary to the disadvantage of CBOW. Herein, our proposed model encodes many linguistic patterns and improves upon sentiment analysis and question classification benchmarks compared to previously reported methods.",
keywords = "continuous bag-of-words, language model, Recurrent neural network, sentence classification",
author = "Lee, {Yo Han} and Kim, {Dong W.} and Lim, {Myo Taeg}",
year = "2019",
month = "2",
day = "1",
doi = "10.1142/S0218213019500027",
language = "English",
volume = "28",
journal = "International Journal on Artificial Intelligence Tools",
issn = "0218-2130",
publisher = "World Scientific Publishing Co. Pte Ltd",
number = "1",

}

TY - JOUR

T1 - A Two-Level Recurrent Neural Network Language Model Based on the Continuous Bag-of-Words Model for Sentence Classification

AU - Lee, Yo Han

AU - Kim, Dong W.

AU - Lim, Myo Taeg

PY - 2019/2/1

Y1 - 2019/2/1

N2 - In this paper, a new two-level recurrent neural network language model (RNNLM) based on the continuous bag-of-words (CBOW) model for application to sentence classification is presented. The vector representations of words learned by a neural network language model have been shown to carry semantic sentiment and are useful in various natural language processing tasks. A disadvantage of CBOW is that it only considers the fixed length of a context because its basic structure is a neural network with a fixed length of input. In contrast, the RNNLM does not have a size limit for a context but only considers the previous context's words. Therefore, the advantage of RNNLM is complementary to the disadvantage of CBOW. Herein, our proposed model encodes many linguistic patterns and improves upon sentiment analysis and question classification benchmarks compared to previously reported methods.

AB - In this paper, a new two-level recurrent neural network language model (RNNLM) based on the continuous bag-of-words (CBOW) model for application to sentence classification is presented. The vector representations of words learned by a neural network language model have been shown to carry semantic sentiment and are useful in various natural language processing tasks. A disadvantage of CBOW is that it only considers the fixed length of a context because its basic structure is a neural network with a fixed length of input. In contrast, the RNNLM does not have a size limit for a context but only considers the previous context's words. Therefore, the advantage of RNNLM is complementary to the disadvantage of CBOW. Herein, our proposed model encodes many linguistic patterns and improves upon sentiment analysis and question classification benchmarks compared to previously reported methods.

KW - continuous bag-of-words

KW - language model

KW - Recurrent neural network

KW - sentence classification

UR - http://www.scopus.com/inward/record.url?scp=85062407832&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85062407832&partnerID=8YFLogxK

U2 - 10.1142/S0218213019500027

DO - 10.1142/S0218213019500027

M3 - Article

VL - 28

JO - International Journal on Artificial Intelligence Tools

JF - International Journal on Artificial Intelligence Tools

SN - 0218-2130

IS - 1

M1 - 1950002

ER -