Segmentation of infant brain MR images is challenging due to insufficient image quality, severe partial volume effect, and the ongoing maturation and myelination processes. In particular, the image contrast inverts around 6-8 months of age, and the white and gray matter tissues are isointense in both T1- and T2-weighted MR images and thus exhibit the extremely low tissue contrast, which poses the significant challenges for automated segmentation. Most previous studies used multi-atlas label fusion strategy, which has the limitation of equally treating the available multi-modality images and is often computationally expensive. In this paper, we propose a novel learning-based multi-source integration framework for infant brain image segmentation. Specifically, we employ the random forest technique to effectively integrate features from multi-source images together for tissue segmentation. The multi-source images include initially only the multi-modality (T1, T2 and FA) images and later also the iteratively estimated and refined tissue probability maps of gray matter, white matter, and cerebrospinal fluid. Experimental results on 119 infant subjects and MICCAI challenges show that the proposed method achieves better performance than other state-of-the-art automated segmentation methods, with significantly reduction of running time from hours to 5 minutes.