Analysis of parallel training algorithms for deep neural networks

Hyowon Lee, Keonnyeong Lee, In Chul Yoo, Dongsuk Yook

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

A majority of training algorithms for deep neural networks (DNNs) use variants of stochastic gradient descent (SGD). Since a large amount of time is typically required to train DNNs, many attempts have been made to speed-up the training process by parallelizing the SGD algorithms. However, such parallelization efforts introduce approximation due to the inherent sequential nature of the SGD methods. In this paper, we revisit and analyze parallel SGD algorithms, and propose a novel pipelined SGD that is more efficient than previous algorithms.

Original languageEnglish
Title of host publicationProceedings - 2018 International Conference on Computational Science and Computational Intelligence, CSCI 2018
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages1462-1463
Number of pages2
ISBN (Electronic)9781728113609
DOIs
Publication statusPublished - 2018 Dec
Event2018 International Conference on Computational Science and Computational Intelligence, CSCI 2018 - Las Vegas, United States
Duration: 2018 Dec 132018 Dec 15

Publication series

NameProceedings - 2018 International Conference on Computational Science and Computational Intelligence, CSCI 2018

Conference

Conference2018 International Conference on Computational Science and Computational Intelligence, CSCI 2018
CountryUnited States
CityLas Vegas
Period18/12/1318/12/15

Keywords

  • Deep neural networks
  • Pipelined stochastic gradient descent
  • Stochastic gradient descent

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Computer Science Applications
  • Hardware and Architecture
  • Information Systems and Management
  • Control and Optimization
  • Modelling and Simulation
  • Artificial Intelligence

Fingerprint Dive into the research topics of 'Analysis of parallel training algorithms for deep neural networks'. Together they form a unique fingerprint.

  • Cite this

    Lee, H., Lee, K., Yoo, I. C., & Yook, D. (2018). Analysis of parallel training algorithms for deep neural networks. In Proceedings - 2018 International Conference on Computational Science and Computational Intelligence, CSCI 2018 (pp. 1462-1463). [8947467] (Proceedings - 2018 International Conference on Computational Science and Computational Intelligence, CSCI 2018). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/CSCI46756.2018.00291