CATS: cache-aware task scheduling for Hadoop-based systems

Byungnam Lim, Jong Wook Kim, Yon Dohn Chung

Research output: Contribution to journalArticle

3 Citations (Scopus)

Abstract

Today with the explosion of big data, data-intensive cluster computing systems have driven to a new data processing paradigm. As Hadoop, one of the most famous data processing frameworks, achieves high performance by running multiple tasks in parallel across nodes in large clusters, task scheduling is considered as one of the most important factors affecting the overall performance. In modern operating systems, caching is used to improve local disk access times, providing data from the main memory without disk accesses. This option, however, is poorly utilized by existing task scheduling methods of Hadoop-based systems, mainly due to the inability of tracking cached data in shared-nothing distributed environments. In this paper, we propose a cache-aware task scheduling method, cache-aware task scheduling (CATS), for Hadoop-based systems which is able to exploit the operating system’s buffer cache and assign tasks to nodes in consideration of the cached data. Through comprehensive experiments, we show that the proposed cache-aware scheduling improves the overall job execution time for various workload types and data sizes.

Original languageEnglish
Pages (from-to)1-15
Number of pages15
JournalCluster Computing
DOIs
Publication statusAccepted/In press - 2017 May 24

    Fingerprint

Keywords

  • Distributed systems
  • Hadoop
  • In-memory
  • Task scheduling

ASJC Scopus subject areas

  • Software
  • Computer Networks and Communications

Cite this