Deep reinforcement learning in continuous action spaces

A case study in the game of simulated curling

Kyowoon Lee, Sol A. Kim, Jaesik Choi, Seong Whan Lee

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Many real-world applications of reinforcement learning require an agent to select optimal actions from continuous action spaces. Recently, deep neural networks have successfully been applied to games with discrete actions spaces. However, deep neural networks for discrete actions are not suitable for devising strategies for games in which a very small change in an action can dramatically affect the outcome. In this paper, we present a new framework which incorporates a deep neural network that can be used to learn game strategies based on a kernel-based Monte Carlo tree search that finds actions within a continuous space. To avoid hand-crafted features, we train our network using supervised learning followed by reinforcement learning with a high-fidelity simulator for the Olympic sport of curling. The program trained under our framework outperforms existing programs equipped with several hand-crafted features and won an international digital curling competition.

Original languageEnglish
Title of host publication35th International Conference on Machine Learning, ICML 2018
EditorsJennifer Dy, Andreas Krause
PublisherInternational Machine Learning Society (IMLS)
Pages4587-4596
Number of pages10
Volume7
ISBN (Electronic)9781510867963
Publication statusPublished - 2018 Jan 1
Event35th International Conference on Machine Learning, ICML 2018 - Stockholm, Sweden
Duration: 2018 Jul 102018 Jul 15

Other

Other35th International Conference on Machine Learning, ICML 2018
CountrySweden
CityStockholm
Period18/7/1018/7/15

Fingerprint

Reinforcement learning
Supervised learning
Sports
Simulators
Deep neural networks

ASJC Scopus subject areas

  • Computational Theory and Mathematics
  • Human-Computer Interaction
  • Software

Cite this

Lee, K., Kim, S. A., Choi, J., & Lee, S. W. (2018). Deep reinforcement learning in continuous action spaces: A case study in the game of simulated curling. In J. Dy, & A. Krause (Eds.), 35th International Conference on Machine Learning, ICML 2018 (Vol. 7, pp. 4587-4596). International Machine Learning Society (IMLS).

Deep reinforcement learning in continuous action spaces : A case study in the game of simulated curling. / Lee, Kyowoon; Kim, Sol A.; Choi, Jaesik; Lee, Seong Whan.

35th International Conference on Machine Learning, ICML 2018. ed. / Jennifer Dy; Andreas Krause. Vol. 7 International Machine Learning Society (IMLS), 2018. p. 4587-4596.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Lee, K, Kim, SA, Choi, J & Lee, SW 2018, Deep reinforcement learning in continuous action spaces: A case study in the game of simulated curling. in J Dy & A Krause (eds), 35th International Conference on Machine Learning, ICML 2018. vol. 7, International Machine Learning Society (IMLS), pp. 4587-4596, 35th International Conference on Machine Learning, ICML 2018, Stockholm, Sweden, 18/7/10.
Lee K, Kim SA, Choi J, Lee SW. Deep reinforcement learning in continuous action spaces: A case study in the game of simulated curling. In Dy J, Krause A, editors, 35th International Conference on Machine Learning, ICML 2018. Vol. 7. International Machine Learning Society (IMLS). 2018. p. 4587-4596
Lee, Kyowoon ; Kim, Sol A. ; Choi, Jaesik ; Lee, Seong Whan. / Deep reinforcement learning in continuous action spaces : A case study in the game of simulated curling. 35th International Conference on Machine Learning, ICML 2018. editor / Jennifer Dy ; Andreas Krause. Vol. 7 International Machine Learning Society (IMLS), 2018. pp. 4587-4596
@inproceedings{f2ed8ec2332e482d80d7bc2158373d03,
title = "Deep reinforcement learning in continuous action spaces: A case study in the game of simulated curling",
abstract = "Many real-world applications of reinforcement learning require an agent to select optimal actions from continuous action spaces. Recently, deep neural networks have successfully been applied to games with discrete actions spaces. However, deep neural networks for discrete actions are not suitable for devising strategies for games in which a very small change in an action can dramatically affect the outcome. In this paper, we present a new framework which incorporates a deep neural network that can be used to learn game strategies based on a kernel-based Monte Carlo tree search that finds actions within a continuous space. To avoid hand-crafted features, we train our network using supervised learning followed by reinforcement learning with a high-fidelity simulator for the Olympic sport of curling. The program trained under our framework outperforms existing programs equipped with several hand-crafted features and won an international digital curling competition.",
author = "Kyowoon Lee and Kim, {Sol A.} and Jaesik Choi and Lee, {Seong Whan}",
year = "2018",
month = "1",
day = "1",
language = "English",
volume = "7",
pages = "4587--4596",
editor = "Jennifer Dy and Andreas Krause",
booktitle = "35th International Conference on Machine Learning, ICML 2018",
publisher = "International Machine Learning Society (IMLS)",

}

TY - GEN

T1 - Deep reinforcement learning in continuous action spaces

T2 - A case study in the game of simulated curling

AU - Lee, Kyowoon

AU - Kim, Sol A.

AU - Choi, Jaesik

AU - Lee, Seong Whan

PY - 2018/1/1

Y1 - 2018/1/1

N2 - Many real-world applications of reinforcement learning require an agent to select optimal actions from continuous action spaces. Recently, deep neural networks have successfully been applied to games with discrete actions spaces. However, deep neural networks for discrete actions are not suitable for devising strategies for games in which a very small change in an action can dramatically affect the outcome. In this paper, we present a new framework which incorporates a deep neural network that can be used to learn game strategies based on a kernel-based Monte Carlo tree search that finds actions within a continuous space. To avoid hand-crafted features, we train our network using supervised learning followed by reinforcement learning with a high-fidelity simulator for the Olympic sport of curling. The program trained under our framework outperforms existing programs equipped with several hand-crafted features and won an international digital curling competition.

AB - Many real-world applications of reinforcement learning require an agent to select optimal actions from continuous action spaces. Recently, deep neural networks have successfully been applied to games with discrete actions spaces. However, deep neural networks for discrete actions are not suitable for devising strategies for games in which a very small change in an action can dramatically affect the outcome. In this paper, we present a new framework which incorporates a deep neural network that can be used to learn game strategies based on a kernel-based Monte Carlo tree search that finds actions within a continuous space. To avoid hand-crafted features, we train our network using supervised learning followed by reinforcement learning with a high-fidelity simulator for the Olympic sport of curling. The program trained under our framework outperforms existing programs equipped with several hand-crafted features and won an international digital curling competition.

UR - http://www.scopus.com/inward/record.url?scp=85057248729&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85057248729&partnerID=8YFLogxK

M3 - Conference contribution

VL - 7

SP - 4587

EP - 4596

BT - 35th International Conference on Machine Learning, ICML 2018

A2 - Dy, Jennifer

A2 - Krause, Andreas

PB - International Machine Learning Society (IMLS)

ER -