Accurate and consistent labeling of longitudinal cortical surfaces is essential to understand the early dynamic development of cortical structure and function in both normal and abnormal infant brains. In this paper, we propose a novel method for simultaneous, consistent, and unbiased labeling of longitudinal dynamic cortical surfaces in the infant brain MR images. The proposed method is formulated as minimization of an energy function, which includes the data fitting, spatial smoothness and temporal consistency terms. Specifically, in the spirit of multi-atlas based label fusion, the data fitting term is designed to integrate adaptive contributions from multi-atlas surfaces, according to the similarity of their local cortical folding with that of the subject surface. The spatial smoothness term is designed to adaptively encourage label smoothness based on the local folding geometries, i.e., also allowing label discontinuity at sulcal bottoms, where the cytoarchitecturally and functionally distinct cortical regions are often divided. The temporal consistency term is further designed to encourage the label consistency between temporal corresponding vertices with similar local cortical folding. Finally, the entire energy function is efficiently minimized by a graph cuts method. The proposed method has been successfully applied to the labeling of longitudinal cortical surfaces of 13 infants, each with 6 serial images scanned from birth to 2 years of age. Both qualitative and quantitative evaluation results demonstrate the validity of the proposed method.