High-speed parallel external sorting of data with arbitrary distribution

Minsoo Jeon, Dong Seung Kim

Research output: Contribution to journalArticle

1 Citation (Scopus)

Abstract

Many parallel sorting algorithms of external disk data have been reported such as NOW-sort, SPsort, hill sort and so on. They all reduce the execution time compared with some known sequential sort; however, they differ in terms of the speed, throughput or cost-effectiveness. Mostly, they deal with uniformly distributed data in their value range. If we divide and redistribute data to processors by fixed and equal division of the key range, all processors will have about equal numbers of keys to sort and store. But if irregularly distributed data are given, the performance will suffer severely as the partitioning would no longer produce balanced loads among processors. Few research results have been reported for parallel external sort of data with arbitrary distribution. In this paper, we develop two distribution-insensitive scalable parallel external sorting algorithms that use sampling technique and histogram counts to achieve even distribution of keys, which eventually contribute to achieve good performance. Experimental results on a cluster of 16 Linux workstations show up to threefold enhancement of the performance compared with NOW-sort for sorting 16 GB integer keys.

Original languageEnglish
Pages (from-to)36-44
Number of pages9
JournalInternational Journal of High Performance Computing and Networking
Volume2
Issue number1
Publication statusPublished - 2004

Fingerprint

Sorting
Cost effectiveness
Throughput
Sampling

Keywords

  • cluster
  • external sort
  • histogram
  • load balancing
  • NOW-sort
  • sample sort

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Software
  • Hardware and Architecture

Cite this

High-speed parallel external sorting of data with arbitrary distribution. / Jeon, Minsoo; Kim, Dong Seung.

In: International Journal of High Performance Computing and Networking, Vol. 2, No. 1, 2004, p. 36-44.

Research output: Contribution to journalArticle

@article{31d034e1ea11445c9692ff27ae580e78,
title = "High-speed parallel external sorting of data with arbitrary distribution",
abstract = "Many parallel sorting algorithms of external disk data have been reported such as NOW-sort, SPsort, hill sort and so on. They all reduce the execution time compared with some known sequential sort; however, they differ in terms of the speed, throughput or cost-effectiveness. Mostly, they deal with uniformly distributed data in their value range. If we divide and redistribute data to processors by fixed and equal division of the key range, all processors will have about equal numbers of keys to sort and store. But if irregularly distributed data are given, the performance will suffer severely as the partitioning would no longer produce balanced loads among processors. Few research results have been reported for parallel external sort of data with arbitrary distribution. In this paper, we develop two distribution-insensitive scalable parallel external sorting algorithms that use sampling technique and histogram counts to achieve even distribution of keys, which eventually contribute to achieve good performance. Experimental results on a cluster of 16 Linux workstations show up to threefold enhancement of the performance compared with NOW-sort for sorting 16 GB integer keys.",
keywords = "cluster, external sort, histogram, load balancing, NOW-sort, sample sort",
author = "Minsoo Jeon and Kim, {Dong Seung}",
year = "2004",
language = "English",
volume = "2",
pages = "36--44",
journal = "International Journal of High Performance Computing and Networking",
issn = "1740-0562",
publisher = "Inderscience Enterprises Ltd",
number = "1",

}

TY - JOUR

T1 - High-speed parallel external sorting of data with arbitrary distribution

AU - Jeon, Minsoo

AU - Kim, Dong Seung

PY - 2004

Y1 - 2004

N2 - Many parallel sorting algorithms of external disk data have been reported such as NOW-sort, SPsort, hill sort and so on. They all reduce the execution time compared with some known sequential sort; however, they differ in terms of the speed, throughput or cost-effectiveness. Mostly, they deal with uniformly distributed data in their value range. If we divide and redistribute data to processors by fixed and equal division of the key range, all processors will have about equal numbers of keys to sort and store. But if irregularly distributed data are given, the performance will suffer severely as the partitioning would no longer produce balanced loads among processors. Few research results have been reported for parallel external sort of data with arbitrary distribution. In this paper, we develop two distribution-insensitive scalable parallel external sorting algorithms that use sampling technique and histogram counts to achieve even distribution of keys, which eventually contribute to achieve good performance. Experimental results on a cluster of 16 Linux workstations show up to threefold enhancement of the performance compared with NOW-sort for sorting 16 GB integer keys.

AB - Many parallel sorting algorithms of external disk data have been reported such as NOW-sort, SPsort, hill sort and so on. They all reduce the execution time compared with some known sequential sort; however, they differ in terms of the speed, throughput or cost-effectiveness. Mostly, they deal with uniformly distributed data in their value range. If we divide and redistribute data to processors by fixed and equal division of the key range, all processors will have about equal numbers of keys to sort and store. But if irregularly distributed data are given, the performance will suffer severely as the partitioning would no longer produce balanced loads among processors. Few research results have been reported for parallel external sort of data with arbitrary distribution. In this paper, we develop two distribution-insensitive scalable parallel external sorting algorithms that use sampling technique and histogram counts to achieve even distribution of keys, which eventually contribute to achieve good performance. Experimental results on a cluster of 16 Linux workstations show up to threefold enhancement of the performance compared with NOW-sort for sorting 16 GB integer keys.

KW - cluster

KW - external sort

KW - histogram

KW - load balancing

KW - NOW-sort

KW - sample sort

UR - http://www.scopus.com/inward/record.url?scp=84951717095&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84951717095&partnerID=8YFLogxK

M3 - Article

VL - 2

SP - 36

EP - 44

JO - International Journal of High Performance Computing and Networking

JF - International Journal of High Performance Computing and Networking

SN - 1740-0562

IS - 1

ER -