Characterizing water quality and quantity profiles with poor quality datin a machine learning algorithm

Zhonghyun Kim, Heewon Jeong, Sora Shin, Jinho Jung, Joon Ha Kim, Seo Jin Ki

Research output: Contribution to journalArticlepeer-review

Abstract

Statistical analyses are often subject to misinterpretation due to poor data quality which is inaccu-rate, incomplete, or unavailable. This study describes how incomplete data diminishes the screening accuracy of water pollution hotspots using a self-organizing map (SOM), a popular algorithm in reducing the dimension of complex data in a nonlinear fashion. A full data set consisting of 12 water quality and quantity parameters monitored monthly over 3 years at the Yeongsan River in Korea was provided to SOM as a reference input. For purposes of comparison, SOM was further allowed to accept three incomplete data sets in terms of variable availability as well as data loss for single and multiple parameters and different pollution levels. We found that data loss of either single or multiple parameters exceeding 15% of the entire data set led to significant changes in spatial and temporal patterns of the original data. However, the variables intentionally unavailable in the given data set affected the screening performance of water pollution hotspots in SOM, to a less obvious extent, as long as the percentage of missing data fell below 10%. The same applied to data loss with three pollution levels, from high through moderate to low concentrations of one important variable. Therefore, we recommend the use of multiple approaches that couple dimensionality reduction algorithms with reasonable imputation methods for the data set with a high percentage (e.g. above 15%) of missing values.

Original languageEnglish
Pages (from-to)127-134
Number of pages8
JournalDesalination and Water Treatment
Volume182
DOIs
Publication statusPublished - 2020 Apr

Keywords

  • Dimensionality reduction
  • Incomplete data
  • Non-linear data analysis
  • Pollution hotspots
  • Self-organizing map
  • Water quality data

ASJC Scopus subject areas

  • Water Science and Technology
  • Ocean Engineering
  • Pollution

Fingerprint Dive into the research topics of 'Characterizing water quality and quantity profiles with poor quality datin a machine learning algorithm'. Together they form a unique fingerprint.

Cite this