The objective of this study is to develop a deformable 3D image registration algorithm for breast PET-CT and MR images based on perturbation-weighted feature information. As a preprocessing, breast regions were separated by regional masks from the abdomen CT and MR. To remove the jig frame that was used for fixing patient posture during the PET-CT scan, we performed the breast segmentation by region growing. Then a point set was extracted from each modality for surface matching. The point set of CT was used as a target model, and the point set of contrast-enhanced MR was used as a reference model. The surface of a breast does not have many features, but we thought that the nipples and scars can be used as salient features to improve registration performance. The proposed algorithm imposes more weights on those features according to the degree of surface perturbation. Furthermore, we used the hierarchical block-wise operation to minimize the error of the local part. After matching the point sets of the global region, we divide the region to N by N blocks and match the point sets of local blocks. We repeat this step by increasing N until the similarity hit the maximum value. Finally, the PET image is transformed with the optimized transformation parameters and is overlaid onto the MR image. The preliminary results show that the surface of breasts from MR and CT matched well, thereby alignment of the clinically meaningful lesions of the MR and PET slices were improved. For future works, we will investigate more features other than surface and improve the performance of algorithm by adjustment of the weights.