The registration of breast DCE-MR images can help correct possible motions during image acquisition, and is also important for diagnosis of breast cancer, i.e., discrimination between benign and malignant tumors. However, deformable registration of DCE-MR images is challenging due to drastic image contrast change over time (especially between pre- and post-contrast images). To improve the registration, we propose a novel hierarchical groupwise registration framework by specially considering the image characteristics of the breast DCE-MR images. Specifically, due to the similarity of post-contrast images, they are jointly registered by a groupwise registration method, and then registered together with the pre-contrast image by a robust correspondence detection technique based on local-steering-kernel features, instead of simple image intensities. Also, to accurately register the tumor region, we treat the motion in tumor area and other normal tissues differently by delineating rigid motion for tumor while non-rigid for other normal tissues. Our experimental results on both real and simulated images show that our method can achieve more consistent and accurate registration results than the conventional pairwise registration method.