Longitudinal atlas construction is a challenging task in medical image analysis. Given a set of longitudinal images of different subjects, the task is how to construct the unbias longitudinal atlas sequence reflecting the anatomical changes over time. In this paper, a novel longitudinal atlas construction framework is proposed. The main contributions of the proposed method lie in the following aspects: (1) Subject-specific longitudinal information is captured by establishing a robust growth model for each subject. (2) The trajectory constraints are enforced for both subject image sequences and the atlas sequence, and only one transformation is needed for each subject to map its image sequence to the atlas sequence while preserving the temporal correspondence. (3) The longitudinal atlases are estimated by groupwise registration and kernel regression, thus no explicit template is used and the atlases are constructed without introducing bias due to the selection of the explicit template. (4) The proposed method is general, where the number of longitudinal images of each subject and the time points at which the images are taken can be different. The proposed method is evaluated on a longitudinal database and compared with a state-of-the-art longitudinal atlas construction method. Experimental results show that the proposed method achieves more consistent spatial-temporal correspondence as well as higher registration accuracy than the compared method.