Resting-state functional magnetic resonance imaging (rs-fMRI) is a non-invasive functional imaging technique that has been widely used to investigate brain functional connectome. Noises and artifacts are dominant in the raw rs-fMRI, making effective noise removal a necessity prior to any subsequent analysis. Without requiring any additional biophysiological recording devices, directly applying independent component analysis on rs-fMRI data becomes a popular process further separating structured noise from signals. However, fast and accurate automatic identification of the noise-related components is critical. Conventional machine learning techniques have been used in training such a classifier with manually engineered features of the components, which usually takes a long time even in the testing phase because its success relies on exhaustively extraction of spatial and temporal features and assembling multiple complicated classifiers to reach satisfactory results. In this paper, we proposed a novel, end-to-end, deep learning-based framework dedicated for noise component identification via effective, automatic, multilayer, hierarchically embedded feature extraction. The merit that does not require any assumptions on the features guarantees its unprecedented performance on the rs-fMRI data even from very heterogeneous cohorts. The speed of this method can be further accelerated due to its inherent ability of parallel computing with GPU. We validate our method with a challenging infant rs-fMRI dataset with high resolution and high quality, which are very different from the commonly used adult data. Our proposed method is more general, hypothesis-free, fast (<1 s for single component classification), and accurate (>97% accuracy).