TY - JOUR
T1 - Simple Yet Effective Way for Improving the Performance of Depth Map Super-Resolution
AU - Yeo, Yoon Jae
AU - Sagong, Min Cheol
AU - Shin, Yong Goo
AU - Jung, Seung Won
AU - Ko, Sung Jea
N1 - Funding Information:
Manuscript received September 29, 2020; revised November 6, 2020; accepted November 15, 2020. Date of publication November 19, 2020; date of current version December 9, 2020. This work was supported by the Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Science, ICT Future Planning, under Grant NRF-2020R1F1A1069009. The associate editor coordinating the review of this manuscript and approving it for publication was Prof. Dezhong Peng. (Corresponding author: Seung-Won Jung.) Yoon-Jae Yeo, Min-Cheol Sagong, Seung-Won Jung, and Sung-Jea Ko are with the Department of Electrical Engineering, Korea University, Seongbuk-gu, Seoul 02841, South Korea (e-mail: yjyeo@dali.korea.ac.kr; mcsagong@dali.korea.ac.kr; swjung83@korea.ac.kr; sjko@korea.ac.kr).
PY - 2020
Y1 - 2020
N2 - In depth map super-resolution (SR), a high-resolution color image plays an important role as guidance for preventing blurry depth boundaries. However, excessive/deficient use of the color image features often causes performance degradation such as texture-copying/edge-smoothing in flat/boundary areas. To alleviate these problems, this letter presents a simple yet effective method for enhancing the performance of the SR without requiring significant modifications to the original SR network. To this end, we present a self-selective concatenation (SSC), which is a substitute for the conventional feature concatenation. In the upsampling layers of the SR network, the SSC extracts spatial and channel attention from both color and depth features such that color features can be selectively used for depth SR. Specifically, the SSC learns to use sufficient color features for rendering sharp depth boundaries, whereas their effects are reduced in smooth regions to prevent texture-copying. The proposed SSC can be included in any existing SR networks that have the encoder-decoder structure. The experimental results show that the proposed method can further improve the performances of existing SR networks in terms of the root mean squared error and peak signal-to-noise ratio.
AB - In depth map super-resolution (SR), a high-resolution color image plays an important role as guidance for preventing blurry depth boundaries. However, excessive/deficient use of the color image features often causes performance degradation such as texture-copying/edge-smoothing in flat/boundary areas. To alleviate these problems, this letter presents a simple yet effective method for enhancing the performance of the SR without requiring significant modifications to the original SR network. To this end, we present a self-selective concatenation (SSC), which is a substitute for the conventional feature concatenation. In the upsampling layers of the SR network, the SSC extracts spatial and channel attention from both color and depth features such that color features can be selectively used for depth SR. Specifically, the SSC learns to use sufficient color features for rendering sharp depth boundaries, whereas their effects are reduced in smooth regions to prevent texture-copying. The proposed SSC can be included in any existing SR networks that have the encoder-decoder structure. The experimental results show that the proposed method can further improve the performances of existing SR networks in terms of the root mean squared error and peak signal-to-noise ratio.
KW - Depth map super-resolution (SR)
KW - convolutional neural network
KW - deep learning
UR - http://www.scopus.com/inward/record.url?scp=85096856956&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85096856956&partnerID=8YFLogxK
U2 - 10.1109/LSP.2020.3039429
DO - 10.1109/LSP.2020.3039429
M3 - Article
AN - SCOPUS:85096856956
VL - 27
SP - 2099
EP - 2103
JO - IEEE Signal Processing Letters
JF - IEEE Signal Processing Letters
SN - 1070-9908
M1 - 9264663
ER -