Various metaheuristic optimization algorithms are being developed and applied to find optimal solutions of real-world problems. Engineering benchmark problems have been often used for the performance comparison among metaheuristic algorithms, and water distribution system (WDS) design problem is one of the widely used benchmarks. However, only few traditional WDS design problems have been considered in the research community. Thus, it is very challenging to identify an algorithm’s better performance over other algorithms with such limited set of traditional benchmark problems of unknown characteristics. This study proposes an approach to generate WDS design benchmarks by changing five problem characteristic factors which are used to compare the performance of metaheuristic algorithms. Obtained optimization results show that WDS design benchmark problems generated with specific characteristic under control help identify the strength and weakness of reported algorithms. Finally, guidelines on the selection of a proper algorithm for WDS design problems are derived.