SSFile: A novel column-store for efficient data analysis in Hadoop-based distributed systems

Jihoon Son, Hyoseok Ryu, Sungmin Yi, Yon Dohn Chung

Research output: Contribution to journalArticle

7 Citations (Scopus)

Abstract

Recently, large-scale relational data analysis has gained much attention. Several Hadoop-based distributed systems have been proposed for scalable relational data analysis. Because the column-store approach is very suitable for analytic queries, many studies on column-oriented storage and query processing for Hadoop-based distributed systems have been conducted. However, two problems have arisen in existing studies, the first of which is that only a small amount of data is processed per task during distributed processing. Each task reads only the necessary data using the columnar structure. Because the task initialization in Hadoop requires a large overhead, it is inefficient that each task processes a small amount of data. The second problem is the lack of support for techniques that optimize columnar execution. Although many such techniques have been proposed for efficient columnar query execution, existing column-store methods for Hadoop-based distributed systems cannot support them efficiently. In this paper, we propose a novel column-store method called SSFile for Hadoop-based distributed systems. SSFile increases the actual amount of data processed per task and supports representative columnar execution techniques for efficient query processing. Through extensive experiments, we show that SSFile significantly improves the performance of distributed processing.

Original languageEnglish
Pages (from-to)68-86
Number of pages19
JournalInformation Sciences
Volume316
DOIs
Publication statusPublished - 2015 Sep 20

    Fingerprint

Keywords

  • Column-store
  • Distributed systems
  • Hadoop
  • HDFS
  • Relational data analysis
  • Server clusters

ASJC Scopus subject areas

  • Artificial Intelligence
  • Software
  • Control and Systems Engineering
  • Theoretical Computer Science
  • Computer Science Applications
  • Information Systems and Management

Cite this