Longitudinal infant studies offer a unique opportunity for revealing the dynamics of rapid human brain development in the first year of life. To this end, it is important to develop tissue segmentation and registration techniques for facilitating the detection of global and local morphological changes of brain structures in an infant population. However, there are two inherent challenges involved in development of such techniques. First, the MR images of the isointense stage - the duration between infantile and early adult stages in the first year of life - have low gray-white matter contrast. Second, temporal consistency cannot be preserved if segmentation and registration are performed separately for different time-points. In this paper, we proposed a 4D joint registration and segmentation framework for serial infant brain MR images. Specifically, a spatial-temporal constraint is formulated to make optimal use of T1 and T2 images, as well as adaptively propagate prior probability maps among time-points. In this process, 4D registration is employed to determine anatomical correspondence across time-points, and also a multi-channel segmentation algorithm, guided by spatial-temporally constrained prior tissue probability maps, is applied to segment the T1 and T2 images simultaneously at each time-point. Registration and segmentation are iterated as an Expectation-Maximization (EM) process until convergence. The infant segmentations yielded by the proposed method show high agreement with the results given by a manual rater and outperform the results when no temporal information is considered.