Cloud-based mapreduce workflow execution platform

In Yong Jung, Byong John Han, Chang-Sung Jeong, Seungmin Rho

Research output: Contribution to journalArticle

3 Citations (Scopus)

Abstract

With increasing demand of data-intensive applications, mapreduce technologies have become useful tools to develop large scale applications efficiently by integrating various existing mapreduce jobs. However, there are few existing researches of workflow systems which can integrates mapreduce jobs with on-demand cloud resource provisioning. In this paper, we present a new cloud-based mapreduce workflow execution platform named DIVE-CWM (Distributed-parallel Virtual Environment on Cloud computing for Workflow for launching Mapreduce jobs) which integrates multiple mapreduce jobs and legacy programs into a single workflow. It provides a transparent and selective job scheduling by estimating the execution time in advance for workflow to execute all its jobs. Also, it supports automatic resource provisioning scheme which offers on-demand VM resources automatically to launch a workflow onto cloud. Furthermore, it provides an agent based resource management for automatic job deployment and execution of workflow on mapreduce clusters. Additionally, service oriented architecture based on web service API and graphical user interface offers high accessibility and convenience to user and other systems. We show the experimental results which compares the different scheduling schemes for various workflows.

Original languageEnglish
Pages (from-to)1059-1067
Number of pages9
JournalJournal of Internet Technology
Volume15
Issue number6
DOIs
Publication statusPublished - 2014 Jan 1

Fingerprint

Scheduling
Launching
Service oriented architecture (SOA)
Cloud computing
Graphical user interfaces
Application programming interfaces (API)
Web services
Virtual reality

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Software

Cite this

Cloud-based mapreduce workflow execution platform. / Jung, In Yong; Han, Byong John; Jeong, Chang-Sung; Rho, Seungmin.

In: Journal of Internet Technology, Vol. 15, No. 6, 01.01.2014, p. 1059-1067.

Research output: Contribution to journalArticle

Jung, In Yong ; Han, Byong John ; Jeong, Chang-Sung ; Rho, Seungmin. / Cloud-based mapreduce workflow execution platform. In: Journal of Internet Technology. 2014 ; Vol. 15, No. 6. pp. 1059-1067.
@article{165b3b49979e47b099315286d67b2879,
title = "Cloud-based mapreduce workflow execution platform",
abstract = "With increasing demand of data-intensive applications, mapreduce technologies have become useful tools to develop large scale applications efficiently by integrating various existing mapreduce jobs. However, there are few existing researches of workflow systems which can integrates mapreduce jobs with on-demand cloud resource provisioning. In this paper, we present a new cloud-based mapreduce workflow execution platform named DIVE-CWM (Distributed-parallel Virtual Environment on Cloud computing for Workflow for launching Mapreduce jobs) which integrates multiple mapreduce jobs and legacy programs into a single workflow. It provides a transparent and selective job scheduling by estimating the execution time in advance for workflow to execute all its jobs. Also, it supports automatic resource provisioning scheme which offers on-demand VM resources automatically to launch a workflow onto cloud. Furthermore, it provides an agent based resource management for automatic job deployment and execution of workflow on mapreduce clusters. Additionally, service oriented architecture based on web service API and graphical user interface offers high accessibility and convenience to user and other systems. We show the experimental results which compares the different scheduling schemes for various workflows.",
keywords = "Cloud computing, Job scheduling, Mapreduce workflow, PaaS",
author = "Jung, {In Yong} and Han, {Byong John} and Chang-Sung Jeong and Seungmin Rho",
year = "2014",
month = "1",
day = "1",
doi = "10.6138/JIT.2014.15.6.17",
language = "English",
volume = "15",
pages = "1059--1067",
journal = "Journal of Internet Technology",
issn = "1607-9264",
publisher = "Taiwan Academic Network Management Committee",
number = "6",

}

TY - JOUR

T1 - Cloud-based mapreduce workflow execution platform

AU - Jung, In Yong

AU - Han, Byong John

AU - Jeong, Chang-Sung

AU - Rho, Seungmin

PY - 2014/1/1

Y1 - 2014/1/1

N2 - With increasing demand of data-intensive applications, mapreduce technologies have become useful tools to develop large scale applications efficiently by integrating various existing mapreduce jobs. However, there are few existing researches of workflow systems which can integrates mapreduce jobs with on-demand cloud resource provisioning. In this paper, we present a new cloud-based mapreduce workflow execution platform named DIVE-CWM (Distributed-parallel Virtual Environment on Cloud computing for Workflow for launching Mapreduce jobs) which integrates multiple mapreduce jobs and legacy programs into a single workflow. It provides a transparent and selective job scheduling by estimating the execution time in advance for workflow to execute all its jobs. Also, it supports automatic resource provisioning scheme which offers on-demand VM resources automatically to launch a workflow onto cloud. Furthermore, it provides an agent based resource management for automatic job deployment and execution of workflow on mapreduce clusters. Additionally, service oriented architecture based on web service API and graphical user interface offers high accessibility and convenience to user and other systems. We show the experimental results which compares the different scheduling schemes for various workflows.

AB - With increasing demand of data-intensive applications, mapreduce technologies have become useful tools to develop large scale applications efficiently by integrating various existing mapreduce jobs. However, there are few existing researches of workflow systems which can integrates mapreduce jobs with on-demand cloud resource provisioning. In this paper, we present a new cloud-based mapreduce workflow execution platform named DIVE-CWM (Distributed-parallel Virtual Environment on Cloud computing for Workflow for launching Mapreduce jobs) which integrates multiple mapreduce jobs and legacy programs into a single workflow. It provides a transparent and selective job scheduling by estimating the execution time in advance for workflow to execute all its jobs. Also, it supports automatic resource provisioning scheme which offers on-demand VM resources automatically to launch a workflow onto cloud. Furthermore, it provides an agent based resource management for automatic job deployment and execution of workflow on mapreduce clusters. Additionally, service oriented architecture based on web service API and graphical user interface offers high accessibility and convenience to user and other systems. We show the experimental results which compares the different scheduling schemes for various workflows.

KW - Cloud computing

KW - Job scheduling

KW - Mapreduce workflow

KW - PaaS

UR - http://www.scopus.com/inward/record.url?scp=84916881661&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84916881661&partnerID=8YFLogxK

U2 - 10.6138/JIT.2014.15.6.17

DO - 10.6138/JIT.2014.15.6.17

M3 - Article

AN - SCOPUS:84916881661

VL - 15

SP - 1059

EP - 1067

JO - Journal of Internet Technology

JF - Journal of Internet Technology

SN - 1607-9264

IS - 6

ER -