The growing collection of longitudinal images for brain disease diagnosis necessitates the development of advanced longitudinal registration and anatomical labeling methods that can respect temporal consistency between images. However, the characteristics of such longitudinal images and how they lodge into the image manifold are often neglected in existing labeling methods. Indeed, most of them independently align atlases to each target time-point image for propagating the pre-defined atlas labels to the subject domain. In this paper, we present a dual-layer groupwise registration method to consistently label anatomical regions of interest in brain images across different time-points using a multi-atlases-based labeling framework. Our framework can best enhance the labeling of longitudinal images through: (1) using the group mean of the longitudinal images of each subject (i.e., subject-mean) as a bridge between atlases and the longitudinal subject scans to align atlases to all time-point images jointly; and (2) using inter-atlas relationship in their nesting manifold to better register each atlas image to the subject-mean. These steps yield to a more consistent (from the joint alignment of atlases with all time-point images) and more accurate (from the manifold-guided registration between each atlases and the subject-mean image) registration, thereby eventually improving the consistency and accuracy for the subsequent labeling step. We have tested our dual-layer groupwise registration method to label two challenging longitudinal brain datasets (i.e., healthy infants and Alzheimer’s disease subjects). Our experimental results have showed that our method achieves higher labeling accuracy while keeping the labeling consistency over time, when compared to the traditional registration scheme (without our proposed contributions). Moreover, the proposed framework can flexibly integrate with the existing label fusion methods, such as sparse-patch based methods, to improve the labeling accuracy of longitudinal datasets.