Magnetic resonance fingerprinting (MRF) is a relatively new imaging framework which allows rapid and simultaneous quantification of multiple tissue properties, such as T1 and T2 relaxation times, in one acquisition. To accelerate the data sampling in MRF, a variety of methods have been proposed to extract tissue properties from highly accelerated MRF signals. While these methods have demonstrated promising results, further improvement in the accuracy, especially for T2 quantification, is needed. In this paper, we present a novel deep learning approach, namely residual channel attention U-Net (RCA-U-Net), to perform the tissue quantification task in MRF. The RCA-U-Net combines the U-Net structure with residual channel attention blocks, to make the network focus on more informative features and produce better quantification results. In addition, we improved the preprocessing of MRF data by masking out the noisy signals in the background for improved quantification at tissue boundaries. Our experimental results on two in vivo brain datasets with different spatial resolutions demonstrate that the proposed method improves the accuracy of T2 quantification with MRF under high acceleration rates (i.e., 8 and 16) as compared to the state-of-the-art methods.