Non-human primates, especially macaque monkeys, with close phylogenetic relationship to humans, are highly valuable and widely used animal models for human neuroscience studies. In neuroimaging analysis of macaques, brain extraction or skull stripping of magnetic resonance imaging (MRI) is a crucial step for following processing. However, the current skull stripping methods largely focus on human brains, and thus often lead to unsatisfactory results when applying to macaque brains, especially for macaque brains during early development. In fact, the macaque brain during infancy undergoes regionally-heterogeneous dynamic development, leading to poor and age-variable contrasts between different anatomical structures, posing great challenges for accurate skull stripping. In this study, we propose a novel framework to effectively combine intensity information and domain-invariant prior knowledge, which are important guidance information for accurate brain extraction of developing macaques from 0 to 36 months of age. Specifically, we introduce signed distance map (SDM) and center of gravity distance map (CGDM) based on the intermediate segmentation results and fuse their information by Dual Self-Attention Module (DSAM) instead of local convolution. To evaluate the performance, we adopt two large-scale and challenging MRI datasets from rhesus macaques and cynomolgus macaques, respectively, with totally 361 scans from two different scanners with different imaging protocols. We perform cross-validation by using one dataset for training and the other one for testing. Experimental results show the robustness of our plug-and-play method on cross-source MRI datasets without any transfer learning.